pashov.net/code|Blog Archives

Author Archives: ivopashov

Coding

Ev(i|a)l code

Published by:

It has been a while since i last saw someone eval code coming from a remote source. I thought that technique is considered evil across the board but i am wrong, many people like it. So i decided to join the cohort of people that preach the perilous use of it.

The context

I have been doing Demandware (Salesforce commerce cloud or SFCC now) for quite some time. They have this custom script built on top of JS called demandware script (.DS). Being a dynamic language, evaling is quite easy as you may guess and a lot of people are tricked into it.

So what was the use case?

In the context of SFCC, you do a lot of object exporting so that you integrate with 3rd party systems. Some integrations take variable count of fields and the customer is the one to decide which fields to export. So how can we give the customer the freedom to decide what to export without going through the hefty procedure of a code change? Eureka! Let them write a bit of a well documented code that we eval and that’s how we construct our exports. If they want an extra field exported, they write an extra line of code that calls some API (either ours or SFCC) that does the magic and the export is ready, brand new and shiny without a single minute of developer time wasted. Customer writing code is as flexible as we get.

La la land

I guess, you buy the flexibility argument until the future you (or someone else) has to deal with the consequences. So why all the grunt:

  • Even though the customer (or whoever that may be) has the flexibility to write code that you execute for her, chances are she will never do that but will come and ask how to do this and that. Say good buy to flexibility.
  • You stop being owner of your code. When someone starts writing code that you eval, you are pretty much opening an API. A vast API. Have you considered the consequences of opening pretty much your whole system as API? Chances are you haven’t. First, you have to be very careful what you eval. Second, say hi to legacy. Forever. You can’t just change code that is being used by external sources. Third, you have to keep a very detailed documentation of who uses your code and how so that you may asses the impact of every code change. The team should be very well aligned with that. And fourth and fifth etc. Not being owner of your code is not cool.
  • Your code stops being greppable and that is very evil. Your code should be absolutely greppable. I have had the unfortunate moment to delete a function that no one uses only to find out later that a ton of stuff is broken because this function is actually used by some external code.
  • Security. Won’t delve here, this is self explanatory.

Is there a valid use case of evaling? It is there for some reason, right?

Yes, of course. I think evaling has some valid use cases and here is my criteria:

  • You should do very simple operations with evaling. Really simple.
  • You should touch as little as you can from the underlying system.
  • You should verify what you eval. Might be a regex, might be something else but you should have a very strict criteria what you eval.

I have done it only once and that’s my use case:

I was doing an application where you could draw windows (like real world windows that you have in your homes) and calculate prices. All that was done client side in the browser in a canvas. So at the end, you have the picture of your window plus the javascript artefacts associated with it. For every window you had the perimeter, area and a bunch of other properties calculated (number of glass panes, area of glass panes etc). However, to come up with a price you should apply a formula. That’s where eval fitted. I gave the customer the possibility to write stuff like that in a text field that i evalled against the constructed window objects:

  •  T + S * 20 (total price calculated by the system + 20 euro per sq. m.)
  • T * 1.2 (20% markup across the board of the total price)
  • T + S * 20 + P * 10 (total price + 20 euro per sq. m. + 10 euro per meter of the perimeter)
  • T + 50 (total price + 50 euro flat per item manufactured)
  • T + GN * 10 (total price + 10 euro per glass pane markup)

You get the point, every merchant does his own way of pricing and that varies a lot. However, all of the pricing formulas are very simple operations and use very limited API (area, perimeter and a couple of more properties of the window). Using a very limited API and doing only arithmetics, there is a very easy way to validate what gets evalled (like with a regex). There is also only one way to calculate area, perimeter and stuff like that in the context of geometrical shape, so you are not stuck with legacy. This code has no reason to change anyways.

TL;DR Don’t do it, unless you have a very concrete use case, open very limited API and can control what will get evalled.

Others

Recommended read – architecture of open source applications

Published by:

Just wanted to share a great resource that I found recently – Architecture of open source applications. These are two books where authors of open source projects share how they built them and the decisions that they took along the way. The last one I read is about Graphite by Chris Davis. Absolutely amazing story and Chris Davis is a very good storyteller. Although I have not built projects that big as Graphite, I have come to the same conclusions on a smaller scale. So you find a lot of these aha moments in the articles.

Enjoy.

Others

Switching from C# to Ruby

Published by:

In March 2016 i switched jobs leaving Progress and joining Receipt Bank. So i had to switch the whole tech stack. In Progress my hands were mainly busy with C#, Web API, Visual Studio, MS SQL, MSMQ and the rest of the MS technologies. In Receipt Bank i we use Ruby (mainly) with Rails as a Web framework. I never actually wanted to switch but it happened this way – i liked the company and had to.

Changing the language is easy. Changing the stack and the tools is not.

When you are a more seasoned programmer, you should be able to switch pain-free. Well, it depends. If you switch from C# to Java or from Ruby to Python it should be fairly easy. It is not that easy if you switch from C# to Ruby as it is not only the language that you will have to learn. You will probably have to drop Windows and adopt Linux or OSX. You will probably have to switch from IDE as VS to a text editor as Sublime, Vim etc. You will most probably use PostgreSQL or MySQL instead of MSSQL. The database change is not that dramatic as the later two being SQL, they share a lot in common with MSSQL. However, dropping the beloved IDE in favor for a text editor is a pain, at least in the beginning. You quickly realise that you may be as productive with a text editor as with an IDE. A power Vim user is at least as productive as a power VS user. The OS change is kind of a dramatic also. I started using OSX and felt great pain in the beginning. Most of my time was in the console which is quite contrary to the Windows experience. However, when you get used to the almighty console, you never look back.

How is the language different?

#1 Ruby is interpreted and C# is compiled. So in Ruby everything is just working until..you hit a problem. C# compiler will complain during compile time if your code is no good.
#2 C# has static typing and Ruby does not. This is not one sided though. Matz (Ruby creator) said that some typing is coming in Ruby 3 and C# introduced var and dynamic to provide for more flexibility. However, the general picture is that Ruby is interested in behaviour and not static properties. This gives great flexibility and i find this rather liberating. To avoid shoot-in-the-leg situations, you have to have very high test coverage with flexible types.
#3 Testing. Flexible languages as Ruby provide for much better testing than C#. In C# you often have one interface with one implementation class which i find stupid. However, you have to do that so you can mock that class in tests. If Ruby you mock pretty everything easily and don’t really need interfaces (there aren’t any also:) ).
#4 Functional programming has much deeper roots in Ruby. C# has made big steps also with LINQ but Ruby is superior in this domain, especially when working with collections.
#5 Ruby has a very powerful feature called blocks. These are like callbacks to functions and are really powerful. There is hardly a class that you can’t find blocks.
#6 Syntax. Ruby is much more minimalistic than C#. Syntax is weird in the beginning but you soon start to love it. Little parens are required and there is no return statement. You can single line a lot of stuff to make code more concise. Syntax is important to write readable code and Ruby definitely has the syntax for that.

What else

I would say that the ruby community is much more vibrant than the C# one. Ruby was created with developer’s happiness as a main goal and developers love it. Ruby is religion for some. Matz is accepted as a rock star in the community. There is nothing like that with the MS technologies.  Giving back to the community is a law for many. That’s why you will find many Ruby developers spend time coaching and teaching others for free. They also contribute to open source big time.

True Ruby developers are committed to quality very much. They also keep code clean and neat with great care. Code must not only work but be beautiful. Fellow developers are very fast to growl if you put one extra coma somewhere. Most of the C# developers i have seen are a bit messy. Quality is also appreciated but readability and code beauty not that much. As i said happiness is a founding principle for the language and you can tell that.

C# is preferred by corporations and Ruby by start ups. So choosing the language will have impact on the company that you will be working in.

Conclusion

I find switching to Ruby a huge win. Despite the pain in the beginning, the language is very elegant and powerful. There are enough good tools also out there to get the job done. The biggest advantage however is the community. Being part of that is guaranteed to higher your level.

 

.NET Coding

Download large files with Web Api as a relay

Published by:

Consider the following scenario: you have a file storage in the cloud (google docs, openstack object storage etc.) and you need to provide your app consumers a way to download the files through your app. The challenges here are mainly memory related – if you get files in memory and then return them in the response, your app will soon be crushed with OutOfmemory Exception. So you need a way to turn your app into a relay which is kind of a pipe through which bytes flow from the file storage to the client without buffering any of the content. I guess you are already thinking streams and you are right.

Change the buffer policy

Web api by default will try to buffer the content that you are trying to download and that needs to be changed. You will need to be inherit from IHostBufferPolicySelector  or inherit from the only implementor of this interface (WebHostBufferPolicySelector) and override its methods :

nbsp;

public class WebHostBufferPolicySelector : IHostBufferPolicySelector
{
    public virtual bool UseBufferedInputStream(object hostContext);
    public virtual bool UseBufferedOutputStream(HttpResponseMessage response);
}

You will also need to override one of the two methods, depending if you want unbuffered content for download or upload. In our case we need UseBufferedOutputStream. A possible implementation would be:

public class NoBufferPolicy : WebHostBufferPolicySelector
    {
        public override bool UseBufferedOutputStream(HttpResponseMessage response)
        {
            if (response.RequestMessage.RequestUri.LocalPath.Contains("download"))
            {
                return false;
            }

            return base.UseBufferedOutputStream(response);
        }
    }

So now we use not buffered content when anybody hits a route that contains download and buffered for any other case. After we have done that we need to register our class in GlobalConfiguration:

 GlobalConfiguration.Configuration.Services.Replace(typeof(IHostBufferPolicySelector), new NoBufferPolicy());

Back to the controller

In the usual case the code in your controller should be something like:

 

var client = new HttpClient();
var responseWithHeadersOnly = await client.GetAsync(requestUrl, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);

Two things are worth mentioning here – the second parameter of the GetAsync method – HttpCompletionOption.ResponseHeadersRead which basically reads only the headers of the response and not the content. The second thing is the ConfigureAwait(false) which has to do with how async and await play in a web app.

Here is the time to check if the response is ok. Once we have the response, we need to get hold of the content stream without putting it in memory:

Stream streamToReadFrom = await responseWithHeadersOnly.Content.ReadAsStreamAsync();

After we get the headers, make sure that everything is ok (no service unavailable, server errors etc.) and have control over the stream it is time to start streaming to the client. We will use a push approach in this case, contrary to the traditional pull. Luckily web api gives us the tools for that – we will use the PushStreamContent object for our response content. What it does is that it reads from a stream and pushes the bytes to the client chunk-wise. It has a couple of overloads but it basically uses a function that takes a Stream, HttpContext and TransportContext as params and may return void or a Task. Let’s see some code:

 var client = new HttpClient();
var streamToReadFrom = await client.GetAsync(requestUrl, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);

                var buffer = new byte[65536];
                var result = Request.CreateResponse();
                int bytesRead = 0;

                result.Content = new PushStreamContent((stream, content, context) =>
                {
                    while ((bytesRead = streamToReadFrom.Read(buffer, 0, buffer.Length)) > 0)
                    {
                        stream.Write(buffer, 0, bytesRead);
                    }

                    stream.Close();
                    streamToReadFrom.Close();
                    client.Displose();
                });

                result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
                result.Content.Headers.ContentDisposition.FileName = filename;
                result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
                return result;

Every time you go through the while loop, you read bytes from the incoming stream and pass them to the outgoing stream.

The flow here is as follows – a request comes in (1), you read the headers of the requested file from the file storage provider (2), return response from the method (return result; line (3)), file starts to download (stream.Write(buffer, 0, bytesRead); (4)), file is done downloading (stream.Close(); (5)). The whole thing ends when you close the outgoing stream and not when you return from the method.

That should be all to turn your web api in a working and scalable file download relay. You can find the source code HERE under the LargeFilesWebApi folder

.NET

Async/Await done right in the context of a web app

Published by:

Since .NET 4.5 we are provided with a handy way for async programming through the await-async operators. However, i feel that these are widely misunderstood as looking at peer’s code.

First things first – what are async-await good for in a web app?

Making your IO bound or computational intensive methods async, makes your app much more scalable. It releases the thread to do any other work (most often serving requests) instead of waiting for computations and IO read/write. If your server can handle 100 people synchronously, it should be ok with at least double that if you use async methods (rule of thumb – depends on many things).

What is a bad idea to use async-await?

If i make a request and that triggers a very slow job, i will be waiting a heck lot of a time until the job is done and the response returned no matter if i use synchronous or asynchronous methods. The benefit of async methods is not that they return fast (you have to await them at the end of the day) but the fact that they utilize threads until doing slow jobs. So response time should be the same but in the mean time the released thread could have server another request before continuing the first job after the await. So if is about utilizing resources and not returning immediately. If you want to return a result to the client immediately and do a job in the background that is a whole new topic. For example, when you make a request to open stack to provision a virtual machine, you get 201 Created immediately, no matter that it takes 10 minutes to provision it. So that is definitely not async-await pattern but rather background workers, queues etc.

Good practice

If you decide to implement async methods go all the way – don’t mix up synchronous and asynchronous code. This may cause deadlock very often. The code below is a deadlock:

[HttpGet]
[Route("api/files")]
public IHttpActionResult Files()
{
    var resultFromLongComputation = this.TestAsync().Result;
    return Ok(resultFromLongComputation);
}

public async Task<int> TestAsync()
{
    await Task.Delay(2000);
    return 42;
}

So we have an async method where we await a long job. When hitting the await line we return immediately in the Files method where we synchronously start to wait for the result from the TestAsync method (the .Result forces the waiting for a Task to complete in synchronous mode). The thread and the related context hang. When the Delay elapses, the TestAsync method wants to get hold of the previous bound context so that it finishes its work and returns a result. However, that context is being blocked by the Files method (it is in a synchronous wait).

How to prevent it?
  1. One way would be to use
public IHttpActionResult Files()
{
    var resultFromLongComputation = await this.TestAsync();
    return Ok(resultFromLongComputation);
}
  2. Another way is to set the ConfigureAwait to false which prevents the method for waiting for the same context to be available. It will grab a thread pool thread instead.
public async Task TestAsync()
{
    await Task.Delay(2000).ConfigureAwait(false);
    return 42;
}

The deadlock is valid for UI apps also (Win forms and WPF) but not for console apps (async in a console app is not bound to the request context or the ui context but just grabs a thread pool thread). So if you have a testing console that is very confusing.

Some words on the Synchronization Context

The default behavior of awaiting a task is to capture the current Synchronization Context and when the task is finished to try to post the invocation of the continuation delegate (compiler generated code that wraps the stuff after the await) on the same Synchronization Context. Synchronization Context will be null in a console app and will be not null in a UI or Web App. So that’s why all the hassle with the contexts and mixing up synchronous and asynchronous code.

Further Reading:

Stephen Cleary intro to async-await

Async/Await FAQ in the pfxteam team msdn blog 

Others

Hacker Podcasts

Published by:

Two of the best hacker podcasts that i have watched.

The first one is by Kevin Mitnick from this year’s Cebit in Germany. He has some amazing demos and stories to tell. It is a little long but definitely worth it.

The second one is by a guy that had his computer stolen. Being a hacker, he could track the thief and find his computer. He also came to know the thief pretty well 🙂

 

Others

Setup Outlook with Gmail

Published by:

Maybe that sounds easy but there are some details that made me struggle for a while so i decided it is worth sharing.

I went ahead to setup a new account on the known screen here in Outlook 2010:

Create account outlook

but i kept getting unauthorized error when trying to go ahead. Here is the trick – if you have setup two way authentication for your account, apps can’t authenticate simply with your credentials but need to be registered in your account and use a generated password. So you should come here: https://security.google.com/settings/security/apppasswords?pli=1 and register your app. As soon as you do so you will get a generated password that you should use in your Outlook client. Here are the other settings:

Pop3 server: pop.gmail.com
Use SSL: Yes
Port: 995

SMTP server: smtp.gmail.com
Use Authentication: Yes
Port for TLS/STARTTLS: 587
Port for SSL: 465

Server timeout: 5 mins instead of the default 1

 

Others

Right IT company for you?

Published by:

I feel that a lot of people (including me) sometimes underestimate the choice of company to work for when that moment comes to pick one. We usually spend 1/3 of our time working and if the match is not close to perfect, dissatisfaction looms quite fast.

So i decided to share some personal impressions on what to be looking for:

Service or product company?

If you work in a service oriented company, you will probably see many projects in short time span which is good as you gather experience fast. You will practice your communication and team skills intensely. However, chances are that the domain will be very similar from project to project – usually companies are specialized in some area or they have repeat customers that require 90% the same stuff as before. Product companies use much larger toolset of technologies, especially if the product is big – you may be doing web, mobile, desktop etc., while in the service company you may be doing only one of the above most of the time. Product companies are also not that tight on deadlines and quality is much higher (they do their own software!). They are usually more stable and rich. Sometimes you will face , however, the five screens problem – you will watch 5 screens for years but that is not that scary as sounds. So in the beginning of your career, service companies are good as you gather experience and confidence fast but if you want to become good software engineer, you should aim at product companies in the long term.

User or enterprise market?

If you want job security, more structured process and probably more money – go with enterprise. That comes with legacy technology, however. Most big companies are reluctant to switch technologies. There are many corporations that use only IE 6-7 so if you work in web, you will have to target those browsers which is a nightmare.

On the other side, if you are in a user oriented company, chances are that you will use latest and greatest. You will have more freedom to experiment, but you will also have to chase trends like crazy. Lagging behind means “death”. So working in that kind of atmosphere may be strenuous, chaotic, unstructured sometimes.

This one is more of a personal preference so i cannot recommend any of the above here.

Big or small?

This one is also matter of preference – but i would suggest you to stick with small companies – up to 50 people. There are some exceptions of course. In a small company, you will see much more of the whole process. You will realize that software companies are not all about writing code. In fact, that is less than 50% of the business process. You will have a chance to communicate with decision makers, managers etc in a direct way which is also a great experience. In a big company, you will probably be part of a team which doesn’t decide much but execute strategies from above.

How to spot good/bad companies?

You can gather great amount of info during the interview with the HR. There you go some signs to watch for and some questions to ask:

  • Do Company HRs know what project they are looking people for? If they don’t – that’s bad sign. Nobody looks just for a software developer, they look for people for specific teams and if the hr doesn’t know that during the interview, that means they didn’t do their homework.
  • If they don’t make you write code or want your Github repo that’s very bad sign. That means you will be working with people that didn’t write code during the interview also. Interviews like “Tell me what are the principles of OOP” are mock interviews the red light should start flashing.
  • Does the team have a QA? If not, that means that the company does value quality that much to pay for a QA. I can tell you that a QA makes a huge difference.
  • Does the company have a practice to write tests? What is the test coverage? Writing tests makes you much better developer.
  • What is the deployment process? If deployment is usually very time consuming and error prone process. If it is not automated, that is very bad sign – that means that the company doesn’t value the time lost deploying.
  • Documentation – is the code documented in any way? I know some of you will say that good code is self-documented but come on. If nobody writes proper docs, being it comments in code, that means that people lack culture of succession.
  • How do people share knowledge within the company? – are there any initiatives as in-house presentations, hackatons etc – very very important point here. As a software developer, knowledge and expertise is your main asset. You want to learn as much as you can during your working day.
  • Does the company have a work home policy? Technologies are developed enough so that software development can be done from anywhere – there is no reason to be 8 hours in office every day. If the company does not allow that , that means that they probably don’t trust you or don’t have a way to monitor your work. Both are bad signs.
  • What is the machine you are working on? What is the chair? Require to sit on the chair even during the first interview. Don’t underestimate that. What is the office like? Make sure that you feel good at the place – you will be spending a lot of time there.
Others

Handy Skype commands

Published by:

I have been using Skype like forever – my user name is ivopashov2006 so you can guess by the name that this makes up about 10 yrs at the time of this writing – but only recently i found out about skype commands, some of which are so handy. Skype commands are meant to serve as utilities or give you some conversation meta data straight by typing the command in the chat area and hitting enter – same as if you were chatting. So those are the ones in my favorites list:

  • /alerstoff – this one stops skype icon in the toolbar flashing when you have new messages in a particular chat group even though you might be in available status. I stay pretty much in available status but i don’t want to get distracted by some funny chats that i am in
  • /alertson [text] – this one takes an optional parameter. If used without parameter, that nullifies the above command and you get all the notifications. However, if used with parameter, that allows you to get notified by specific words. So /alertson food will notify you only if someone mentions food.
  • /remotelogout – that is valid only for cloud based chats (most are like that) and signs you out from all other instances that you are logged in except the current one. I use this to double check if i signed out from the office skype without touching the remote desktop.
  • /add [skypename] – super easy way to add someone to the chat if you know the username
  • /leave – drop off from a conversation
  • /showmembers – this is handy for group chats – it shows all members in the chat with their roles
  • /get role – gets you role in the chat
  • /help – this is pretty much self explanatory

There are of course some more but these are the ones that you will probably be using 90% of the time – for the rest refer to /help.