LombaXMonday N° 8 – links and resources from the IT World

What is this? Every day I read a lot of articles about computer programming and languages, infrastructures, new technologies, quick tips, trends, architectures and so on. I read them immediately or put in my “read later” list. All my readings helped me to grow and learn new things…so why don’t share them with my readers? For this reason, I decided to publish and share, every monday, a list of the links I came accross during the past days.


Linux quick tips & commands for performance monitoring

Here is my list of commands that I use to understand why a Linux machine is hanging, running slow or having unexpected behavior

Process analysis, load average, wa, cpu/ram used, disk usage
top
htop
iotop

Process tree, uninterruptible sleep process (D)
ps auxf –> D state processes are in uninterruptible sleep
ps axl –> under the WCHAN column you can see on which kernel function uninterruptible sleep processes are stuck

Stats over time
vmstat 1
iostat 1

Network
netstat
ss -l (network queue length)

Tracing
lsof
strace -e trace=open <application> <— trace an application
strace -e trace=open -p <pid> <— trace a pid

 


Ultra fast http server for local file sharing

python -m http.server 7777

warning: this command shares the content of the current directory via the choosen port (in this case 7777) to the WHOLE WORLD. Use carefully and remember to close it when you have finished


Bash parameter expansion in depth – link

An useful article that explains in depth how parameter expansion works in bash


That’s all for today’s LombaXMonday, if you liked it, have any question or want to let me know that this article sucks, don’t hesitate to add a comment 🙂

LombaXMonday N° 7 – links and resources from the IT World

What is this? Every day I read a lot of articles about computer programming and languages, infrastructures, new technologies, quick tips, trends, architectures and so on. I read them immediately or put in my “read later” list. All my readings helped me to grow and learn new things…so why don’t share them with my readers? For this reason, I decided to publish and share, every monday, a list of the links I came accross during the past days.


Quickstart with Elasticsearch – link

I always wanted to try Elasticsearch but the time is never enough. I found this 5 minutes tutorial that is a good start point.

One quicker variant is to use the elasticsearch docker image to do all the tests. At this link you will find how, shortly:

docker pull docker.elastic.co/elasticsearch/elasticsearch:5.4.1

docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" docker.elastic.co/elasticsearch/elasticsearch:5.4.1

Another thing to do, in all the example curl commands of the provided links you must add the authentication. For example, the command

curl -XPUT 'http://localhost:9200/blog/user/dilbert' -d '{ "name" : "Dilbert Brown" }'

must be changed in

curl --user elastic:changeme -XPUT 'http://localhost:9200/blog/user/dilbert' -d '{ "name" : "Dilbert Brown" }'

adding –user elastic:changeme (default user-password of the Elasticsearch docker image).
Enjoy


Chain of Responsibility Pattern – link

and why is different from the Decorator Pattern: link and link


Comment one line programmatically with sed

Quick tip: suppose we have this line in /etc/php5/cgi/php.ini file

disable_functions = pcntl_alarm,pcntl_fork

if we want to comment this line programmatically, simply do

sed -e '/disable_functions/ s/^;*/;/' -i /etc/php5/cgi/php.ini

 


That’s all for today’s LombaXMonday, if you liked it, have any question or want to let me know that this article sucks, don’t hesitate to add a comment 🙂

LombaXMonday N° 6 – links and resources from the IT World

What is this? Every day I read a lot of articles about computer programming and languages, infrastructures, new technologies, quick tips, trends, architectures and so on. I read them immediately or put in my “read later” list. All my readings helped me to grow and learn new things…so why don’t share them with my readers? For this reason, I decided to publish and share, every monday, a list of the links I came accross during the past days.


JQ, an useful tool for handling JSON – link and link

JQ is a very useful command line tool for handling JSON.
It permits you to extract parts of a JSON using a simple syntax.

Look at this example

If you want to take the Title and the Acronym (and change their name):

curl 'https://www.lombax.it/documents/json.json' | jq '.glossary | {MyTitle: .title, MyAcronym: .GlossDiv.GlossList.GlossEntry.Acronym}'

Explore utf8mb4 performance in the new MySQL 8.0 (not yet released) – link

MySQL 8.0 is in the last phases of the development process and it brings a lot of new features. One of this is an huge increase in performances even using the news utf8mb4 encoding.


HTTPS on StackOverflow – link

Implementing HTTPS on a website may seem simple and straightforward. But what happens when you manage the world biggest Q&A site with hundreds of domains?
Enjoy this (long but good) post by Nick Craver


That’s all for today’s LombaXMonday, if you liked it, have any question or want to let me know that this article sucks, don’t hesitate to add a comment 🙂

LombaXMonday N° 5 – links and resources from the IT World

What is this? Every day I read a lot of articles about computer programming and languages, infrastructures, new technologies, quick tips, trends, architectures and so on. I read them immediately or put in my “read later” list. All my readings helped me to grow and learn new things…so why don’t share them with my readers? For this reason, I decided to publish and share, every monday, a list of the links I came accross during the past days.


Inspect and inject HTTP/HTTPS requests with mitmproxy – link

Mitmproxy is an useful tool for web developers. It permits to inspect web requests (https requests to) acting as a middleware between your browser and the final url. Yes, modern browser inspectors gives us the inspection functionalities without installing anything, but mitmproxy permits you to block/pause the requests and inject/change the content manually. You can add filters on specific words/patterns in the request (url, cookies and so on), and then mitmproxy will block the request and wait for your input. Then, you will be able to change the request content. This is useful expecially when debugging complex request/response flows: you will be able to do all the flow via browser, and then add a “breakpoint” only when needed. Cool


Cloud Computing comes at a price – link

Short story of a startup that forgot that “pay per use” is not always as cheap as it seems…expecially if you don’t make correct planning


That’s all for today’s LombaXMonday, if you liked it, have any question or want to let me know that this article sucks, don’t hesitate to add a comment 🙂

LombaXMonday N° 4 – links and resources from the IT World

What is this? Every day I read a lot of articles about computer programming and languages, infrastructures, new technologies, quick tips, trends, architectures and so on. I read them immediately or put in my “read later” list. All my readings helped me to grow and learn new things…so why don’t share them with my readers? For this reason, I decided to publish and share, every monday, a list of the links I came accross during the past days.


Cars and satellite insurance, how to do it (wrong) – link

Andrea Scarpino tell us how he discovered a big security issue in the API provided by his insurance company. With a little bit of reverse engineering on the satellite device provided by the company, he was able to retrieve all the personal informations of all the insurance customers.


Interactice shell scripts with expect command – link

Everyone have tried, at least once, to automate a task writing a simple bash script.
However, how to handle a prompt (for example a password prompt?).

`expect` is a simple program that, with an easy-to-learn syntax permits you to write scripts that expect for a specific prompt before sending something back


MySQL multiple keys and indexes, did you know? by Enomis-

Let’s assume that we have this simple table with an huge quantity of data:

CREATE TABLE test (
id INT NOT NULL,
last_name CHAR(30) NOT NULL,
first_name CHAR(30) NOT NULL,
PRIMARY KEY (id),
INDEX name (last_name,first_name)
);

as you can see, we have an index on last_name, first_name

Now, try to query it:

select * from test where first_name=’Fabio’ and last_name=’Lombardo’; // time: 0.5sec

select * from test where last_name=’Lombardo’ and first_name=’Fabio’; // time: 0.01sec

Why this difference?
In MySQL, multiple indexes must be queried in the exact order they have been declared.

More informations:

Multiple-Column Indexes
Avoiding Full Table Scans
Index Hints


That’s all for today’s LombaXMonday, if you liked it, have any question or want to let me know that this article sucks, don’t hesitate to add a comment 🙂

LombaXMonday N° 3 – links and resources from the IT World

What is this? Every day I read a lot of articles about computer programming and languages, infrastructures, new technologies, quick tips, trends, architectures and so on. I read them immediately or put in my “read later” list. All my readings helped me to grow and learn new things…so why don’t share them with my readers? For this reason, I decided to publish and share, every monday, a list of the links I came accross during the past days.


Check the technology behind a website. Is it WordPress? Is it Joomla? – link

Do you want to know what technology power up a website? Tired of inspecting html and headers for clues? Is it WordPress? Is it Joomla? Don’t worry, simply paste the URL in the provided link and enjoy the results


ES6 Overview In 350 Bullet Points – link

A quick and easy recap of all the new features and syntax changes of ES6


Testing code that emits output and native functions in PHP – link

You know (and if you don’t, you should!!! :-), the scope of TDD is to test pieces of software to avoid that it breaks when you change something. It’s very useful because you can simply “run tests” and be sure (if you wrote and structured them correctly) that your changes didn’t affect anything. However, writing good tests is not as simple as it seems, and TDD have various type of tests (for example Unit Tests, Integration Tests, Behavior Tests and so on).

In this link we focus on Unit Tests. In Unit Tests, the difficult part is to decouple things and test all classes as separate, independent entities and test them against their interfaces. You test that, when the contract is respected (when you call their method with the correct dependencies), the class behaves as expected.

To create a successfull Unit Test, it’s very important to test objects only against their interfaces and to mock (see Mockery for php, for example) all their dependencies. With mocks, you create fake dependencies (objects) and check the correct sending/receiving on messages by/to the tested class. You focus mainly on the “communication” between the tested object and the rest of the world. With mocks, you can test a single object without having to integrate all the things together (for example, you can test a query builder without having to use a real database during the test phase).

However, when it comes to native php functions, it seems difficult to create mocked functions and objects. With this guide, you will learn some useful trick and Php language features that will help you in the building of your Unit Tests.


 

That’s all for today’s LombaXMonday, if you liked it, have any question or want to let me know that this article sucks, don’t hesitate to add a comment 🙂

LombaXMonday N° 2 – links and resources from the IT World

What is this? Every day I read a lot of articles about computer programming and languages, infrastructures, new technologies, quick tips, trends, architectures and so on. I read them immediately or put in my “read later” list. All my readings helped me to grow and learn new things…so why don’t share them with my readers? For this reason, I decided to publish and share, every monday, a list of the links I came accross during the past days.


Monitor your AWS CloudTrail events with Slack and GorillaStack – link

Although it is based on a commercial product, at the provided link you can find a free Slack Bot that will monitor all your AWS CloudTrail events and alert you when something happen. Personally,  I added some triggers to increase the security of our AWS Account: I get an alert when a user login to the AWS Console, when an AWS EC2 Instance is created, rebooted or terminated. It is very easy to install: after adding it to slack, a Cloudformation template that creates everything will be provided, basically an IAM Role, an S3 Bucket, a Lambda function and all the needed CloudTrail configurations are created automatically with a simple step-by-step procedure.


Expose your local development website over a public URL – link

So, you have created a brand new website and you want to show to your customer. But you haven’t yet deployed it to the hosting provider, it’s only available on your local machine via http://localhost. No worries, with ngrok you will be able to share it immediately, via a public URL, writing a simple command. No need to setup a VPN, and event supporting SSL.

Simply install it, write this in your terminal

ngrok http 80

and your site will be publicly available on an URL similar to this: http://xyza335.ngrok.io


What is CQRS? – link and link

Martin Fowler gives us a brief explanation of what is CQRS and its benefits. By splitting the model in two parts (one dedicated to the **query** and the other dedicated to the **command** part) structured applications with complex domains can have huge benefits. But be aware that all comes to a price…don’t over engineer 🙂


 

That’s all for today’s LombaXMonday, if you liked it, have any question or want to let me know that this article sucks, don’t hesitate to add a comment 🙂

LombaXMonday N° 1 – links and resources from the IT World

Hello there! With this post I want to inaugurate a new section of this blog: LombaXMonday!
One of the things that my wife complains about me is the fact that I don’t read books. However, it doesn’t mean that I don’t read. I’m an avid reader of news of all kinds regarding technology, programming and IT.
Every day I read a lot of links about computer programming and languages, infrastructures, new technologies, quick tips, trends, architectures and so on. I read them immediately or put in my “read later” list. All my readings helped me to grow and learn new things…so why don’t share them with my readers? For this reason, I decided to publish and share, every monday, a list of the links I came accross during the past days.

Ok, now that I explained why, let’s begin with the first LombaXMonday


A roadmap to becoming a web developer in 2017 – link

This article suggests a learning path to become a web developer in 2017. It differentiates between frontend, backend and devops. Despite its content can be debatable, you can give it a look even if you are an experienced web developer, so you can evaluate if you lacks of some nice-to-have skills and start learning again.


Composition over Inheritance – link

Ok, many of you knows, I don’t like Javascript and the frontend in general 🙂
However, I think that the concepts explained in this video are useful also for backend developers and can be easily ported to the PHP ecosystem. Learn how to create your objects using Composition and avoid the headaches of the classic Inheritance.


The art of command line (Italian) – link

Do you think to be a master of the command line? Do you think that Bash has no secrets for you? You are wrong. For sure. Did you know the ctrl-r shortcut? And ctrl-w?
Spend a little time studying the linked article and learn how to master – REALLY – the command line. You will never stop to use these new tips.
Note: The article is in Italian


 

That’s all for the first LombaXMonday, if you liked it, have any question or want to let me know that this article sucks, don’t hesitate to add a comment 🙂

Soluzioni per Live Streaming

NB: questo articolo è stato scritto velocemente, ma vuole riassumere diverse giornate di analisi delle tecnologie e del mercato.
Se non capite qualche parte o volete maggiori dettagli, contattatemi pure. Con il tempo aggiungerò dettagli.

Parliamo delle tecnologie di Live Streaming, sistemi per trasmettere live e a migliaia di persone, in tempo reale, un flusso video.

Ho effettuato questa analisi per esigenza aziendale.

Nel nostro caso l’esigenza era duplice: in alcuni casi, trasmettere un evento live a qualche migliaio di persone (troupe video presente). In altri casi, effettuare brevi trasmissioni da parte di singoli (responsabili dei ns clienti), non assistiti (quindi tecnologia *semplice*), a diverse centinaia di persone.

Prima di entrare nei dettagli, è bene analizzare le varie parti dello stack di cui un Live Streaming si compone


 

Fase di ripresa e acquisizione (transcoding):

Questa prima fase è, banalmente, il momento in cui effettuate le riprese. Dato che parliamo di “Live”, il flusso video deve essere catturato e trasmesso in tempo reale. Qualunque sia il vostro sistema video per le riprese (telecamere professionali, webcam, cellulari), il vostro scopo è quello di far arrivare un segnale video a un pc, che dovrà poi codificarlo secondo uno standard e trasmetterlo a un server.

Esistono inoltre delle App per cellulari che facilitano il compito: acquisiscono il segnale dalla telecamera on-board, lo impacchettano e spediscono direttamente secondo lo standard.
Per la fase di trasmissione, ci focalizziamo sullo standard RTMP ( https://it.wikipedia.org/wiki/Real_Time_Messaging_Protocol ). E’ possibile usare anche RTSP ( https://it.wikipedia.org/wiki/Real_Time_Streaming_Protocol ). Non ho analizzato altri standard.


 

Fase di creazione dello stream:

Questa fase viene effettuata da un server dedicato, quello a cui state inviando il flusso RTMP. Il server riceve il flusso video, lo transcodifica ulteriormente (producendo i chunk video a varie risoluzioni/bitrate e formati) e li rende disponibili per la fruizione.


 

Fase di distribuzione:

il server che crea lo stream non ha una banda sufficiente a servire tutti i vostri utenti. Per questo motivo non viene esposto direttamente su internet, viene piuttosto utilizzato un sistema di distribuzione (tipicamente una CDN https://it.wikipedia.org/wiki/Content_Delivery_Network ). Quindi il vostro server di encoding carica lo stream definitivo sulla CDN rendendolo disponibile agli utenti


 

Fase di fruizione:

i vostri utenti tipicamente (ma non necessariamente) si collegano al sito internet dove ospitate il video. In questo sito è inserito un player, che tipicamente è un’applicazione Flash o un player HTML5. All’avvio della fruizione, il player contatta l’url CDN e inizia a scaricare il video. Anche in questa fase, esistono diversi standard e tecnologie: quelle che ho analizzato sono sempre RTMP (può essere usato indipendentemente sia per la fase di upload che per quella di download), oppure HLS (standard più recente)
Una variabile importante, se lavorate su siti internet, è la possibilità di vedere il video direttamente dalla pagina che lo ospita (embed), senza farlo aprire in un popup. Altra variabile importante è la possibilità di scegliere diversi bitrate manualmente o in automatico (a seconda della banda disponibile lato client). In questo caso parleremo di JWPlayer ( https://www.jwplayer.com ) che gestisce tutte queste casistiche.


 

Ora che abbiamo parlato delle varie parti dello stack, passiamo a un po’ di nomi, tecnologie e consigli. Iniziamo con il fai da te, se volete dei servizi che coprono tutta questa parte in pieno outsourcing staltate pure questa parte. Anche se vi consiglio caldamente di leggerla per imparare qualcosa 😉

Per la fase di ripresa
possiamo utilizzare telecamere professionali con uscita analogica o digitale, acquisendo poi il segnale con una scheda di acquisizione tipo questa:

http://www.amazon.it/Blackmagic-Design-Intensity-Shuttle-Thunderbolt/dp/B007ZDHDRS

Oppure è possibile usare direttamente la webcam.
In questa fase vogliamo encodare il segnale in ingresso in un flusso RTMP da inviare al nostro server.
Esistono diversi software per questa fase, io personalmente consiglio Wirecast: http://www.telestream.net/wirecast/overview.htm

Il setup è semplice, una volta acquisito correttamente il segnale video dovrete impostare i dati del server RTMP che riceverà il flusso video. Tipicamente saranno un indirizzo ip, uno username e una password.

Se volete un setup più semplice per eseguire le prime prove, scaricate un’app all-in-one, su iPhone posso consigliare questa: https://itunes.apple.com/it/app/broadcast-me/id491982406?mt=8 (esiste anche una versione RTSP, ma in questo caso noi vogliamo usare la versione RTMP)

per la fase di creazione dello stream:
il nostro server riceverà lo stream RTMP e dovrà encodarlo secondo i diversi standard che vogliamo poi fornire al nostro player (passando per la CDN).
Gli strumenti consigliati in questa fase sono due:
*Apple Media Encoder e altri software a supporto (ostico ma sicuramente il più configurabile)
*Adobe Media Encoder (non l’ho usato personalmente)

In questa fase consiglio di creare uno stream HLS ( https://en.wikipedia.org/wiki/HTTP_Live_Streaming ), in particolare l’encoder prende il video live e lo segmenta in piccoli pezzi (da 10 secondi ciascuno), in tante risoluzioni diverse, e li carica sulla CDN. L’encoder mantiene aggiornato un file di index (file .m3u8) che viene usato dal player per referenziare correttamente i file man mano che vengono aggiunti.

per la fase di distribuzione:

in questa fase ci servirà appoggiarci a una CDN e depositare i file sullo storage ad essa associato. In alcuni casi sarà la CDN a recuperare i file (tramite una http request a un vostro server, anche direttamente al media server se volete).
Poco da dire, facciamo qualche nome:
• Akamai
• CDNetworks
• Edgecast
• Limelight

ce ne sono tante altre, ma queste hanno un particolare: possono ricevere (e poi redistribuire) uno stream RTMP direttamente, senza la necessità di un encoder intermedio come da punto due. In questa guida non copriamo questa casistica, ma è giusto saperlo. Io comunque vi sconsiglio questo setup, in quanto RTMP è superato per la parte di distribuzione, in favore del più recente e flessibile HLS.

per la fase di fruizione:

anche in questo caso poche parole, vi consiglio di usare JWPlayer che ha già il supporto nativo per HLS e un player HTML5 nativo (o flash in fallback nel caso il vostro device non supporti html5).
A questa pagina trovate dettagli per il setup: https://support.jwplayer.com/customer/portal/articles/1430218-using-hls-streaming


 

Una volta parlato del fai da te, è giusto dire che non è sempre corretto reinventare la ruota. Esistono servizi che, a fronte di un costo mensile, coprono in tutto o in parte le varie parti dello stack.

Il più completo che ho trovato è Ustream: copre l’intero stack. Fornisce un’interfaccia web per l’acquisizione, con la possibilità di acquisire dalla webcam direttamente all’interno del browser e, nella versione pro, fornisce il codice embed per il player. Insomma, copre tutte e 4 le fasi in maniera molto semplice.
Si possono utilizzare telecamere professionali ed effettuare lo streaming anche tramite un software dedicato, Ustream producer. Inoltre vi permette di inviare il vostro flusso RTMP direttamente (se volete usare un’app o uno streamer hardware).

Altri nomi di cui scriverò più avanti: DaCast, StreamZilla.


 

Vi lascio alcuni link utili:
Bibbia dello streaming video by Apple

Strumenti server Apple

Alcuni transcoder:
https://zencoder.com/en/hls
https://www.wowza.com/forums/content.php?304-How-to-set-up-and-run-Wowza-Transcoder-for-live-streaming#swReq
http://knowledge.kaltura.com/live-streaming-using-kaltura-live-streaming-hds-hls-dash

Alcune app per streaming da iPhone
http://www.dacast.com/blog/how-to-stream-live-from-iphone/
https://itunes.apple.com/it/app/broadcast-me/id491982406?mt=8
https://itunes.apple.com/it/app/os-broadcaster/id632458541?mt=8