I am interested in on how to compute stuff on my GPU, for a hobby project I am currently working it. Previously, I learned a bit about OpenGL, while writing my own (bad) renderer for Gaussian Splatting, so I wanted to learn how to do it with OpenGL.

So, I found this great tutorial about compute shaders in opengl 4.3 by dan coady. It explains how to use compute shaders in OpenGL, but instead of shader storage buffer objects (SSBOs) textures are used. That works nicely but one drawback is that textures are more limited in their size than a SSBO. For example on my machine a SSBO can be up to 1GB big, while for a texture I can only have up to 64MB (16384 pixels with RGBA = 16384 * 4 bytes).

As I am missing the endurance to write a detailed in-depht blog article (I’m sorry), I’ll let the code below speak for itself. I have divided it into 9, hopefully self explaining, parts, that describe how to use compute shaders and shader storage buffer objects in OpenGL to do calculations on your GPU instead of your CPU.

Code

You can find the complete source code here.

Part 1: Set up OpenGL

We need a context for OpenGL, this is mandatory. Normally we set up a window, but in our case we actually don’t want to render/display anything, so we directly hide the window. That works for when running the code on your computer, but requires a display server like X11 or Wayland. That may be not the case if you want to execute this on a server. However, that should also be possible if you use EGL instead of glfw to write a headless version, which can run even if no graphical interface exists.

 1fn main() {
 2    // Set up context for OpenGL
 3    let mut glfw = glfw::init(glfw::fail_on_errors).unwrap();
 4    glfw.window_hint(glfw::WindowHint::ContextVersionMajor(4));
 5    glfw.window_hint(glfw::WindowHint::ContextVersionMinor(6));
 6    glfw.window_hint(glfw::WindowHint::OpenGlProfile(
 7        glfw::OpenGlProfileHint::Core,
 8    ));
 9
10    glfw.window_hint(glfw::WindowHint::FocusOnShow(true));
11    glfw.window_hint(glfw::WindowHint::Visible(false));
12
13    let (mut window, _) = glfw
14        .create_window(256, 256, "Compute shaders SSBO Example", glfw::WindowMode::Windowed)
15        .expect("Failed to create GLFW window.");
16
17    // Initialize OpenGL
18    gl::load_with(|symbol| window.get_proc_address(symbol) as *const _);
19    
20    // [...]
21}

Part 2: Compile the shader

 1fn main() {
 2    // [...]
 3
 4    let shader = compile_shader(Path::new("src/compute_shader.glsl"));
 5    let program = link_shader(shader);
 6    
 7    // [...]
 8}
 9
10fn compile_shader(path: &Path) -> gl::types::GLuint {
11    let shader;
12
13    let code = fs::read_to_string(path).expect("could not read shader file");
14    let code = CString::new(code).unwrap();
15
16    unsafe {
17        shader = gl::CreateShader(gl::COMPUTE_SHADER);
18        gl::ShaderSource(shader, 1, &code.as_ptr(), ptr::null());
19        gl::CompileShader(shader);
20
21        // [...] omit error handling for simplicity
22    }
23
24    shader
25}
26
27fn link_shader(shader: gl::types::GLuint) -> gl::types::GLuint {
28    unsafe {
29        let program = gl::CreateProgram();
30        gl::AttachShader(program, shader);
31        gl::LinkProgram(program);
32
33        let mut success = gl::FALSE as gl::types::GLint;
34        gl::GetProgramiv(program, gl::LINK_STATUS, &mut success);
35
36        // [...] omit error handling for simplicity
37
38        gl::DeleteShader(shader);
39
40        program
41    }
42}

Part 3: Our compute shader

 1#version 460 core
 2
 3layout(local_size_x = 1, local_size_y = 1, local_size_z = 1) in;
 4
 5layout(std430, binding = 1) readonly buffer input_0_buffer {
 6    uint input_0[];
 7};
 8
 9layout(std430, binding = 2) readonly buffer input_1_buffer {
10    uint input_1[];
11};
12
13layout(std430, binding = 3) writeonly buffer output_buffer {
14    uint data[];
15};
16
17uniform uint factor;
18
19void main() {
20    uint index = gl_GlobalInvocationID.x;
21
22    data[index] = input_0[index] * input_1[index] * factor;
23}

Part 4: Set up the shader storage buffer objects

 1fn main() {
 2    // [...]
 3
 4    // Set up our shader storage buffer objects
 5    let size = 10;
 6
 7    let input_1_ssbo = setup_ssbo(1, size * size_of::<u32>(), gl::DYNAMIC_DRAW);
 8    let input_2_ssbo = setup_ssbo(2, size * size_of::<u32>(), gl::DYNAMIC_DRAW);
 9    let output_ssbo = setup_ssbo(3, size * size_of::<u32>(), gl::STREAM_COPY);
10    
11    // [...]
12}
13
14fn setup_ssbo(
15    index: gl::types::GLuint,
16    size: usize,
17    usage: gl::types::GLenum,
18) -> gl::types::GLuint {
19    let mut ssbo = 0;
20
21    unsafe {
22        gl::GenBuffers(1, &mut ssbo);
23        gl::BindBuffer(gl::SHADER_STORAGE_BUFFER, ssbo);
24
25        gl::BufferData(
26            gl::SHADER_STORAGE_BUFFER,
27            size as gl::types::GLsizeiptr,
28            ptr::null(),
29            usage,
30        );
31        gl::BindBufferBase(gl::SHADER_STORAGE_BUFFER, index, ssbo);
32
33        gl::BindBuffer(gl::SHADER_STORAGE_BUFFER, 0);
34    }
35
36    ssbo
37}

Part 5: Write data to the GPU’s shader storage buffer objects

 1fn main() {
 2    // [...]
 3
 4    // Write data to the GPU
 5    let input_1_data: Vec<u32>= vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
 6    update_ssbo_data(1, &input_1_data, 0);
 7
 8    let input_2_data: Vec<u32>= vec![41, 42, 43, 44, 45, 46, 47, 48, 49, 50];
 9    update_ssbo_data(2, &input_2_data, 0);
10    
11    // [...]
12}
13
14fn update_ssbo_data(
15    index: gl::types::GLuint,
16    data: &Vec<u32>,
17    offset: usize,
18) {
19    let data_size = size_of::<u32>() * data.len();
20
21    unsafe {
22        gl::BindBuffer(gl::SHADER_STORAGE_BUFFER, index);
23
24        gl::BufferSubData(
25            gl::SHADER_STORAGE_BUFFER,
26            offset as gl::types::GLsizeiptr,
27            data_size as gl::types::GLsizeiptr,
28            data.as_ptr() as *const std::ffi::c_void,
29        );
30
31        gl::BindBuffer(gl::SHADER_STORAGE_BUFFER, 0);
32    }
33}

Part 6: Set some static GPU’s uniform data

 1fn main() {
 2    // [...]
 3
 4    // Set some static data
 5    let factor = 4;
 6    set_uniform("factor", factor, program);
 7    
 8    // [...]
 9}
10
11fn set_uniform(name: &str, value: u32, program: gl::types::GLuint) {
12    let uniform_name = CString::new(name).unwrap();
13
14    unsafe {
15        gl::UseProgram(program);
16
17        let uniform_location = gl::GetUniformLocation(program, uniform_name.as_ptr());
18
19        gl::Uniform1ui(uniform_location, value as gl::types::GLuint);
20    }
21}

Part 7: Compute and wait for the results

 1fn main() {
 2    // [...]
 3
 4    // Compute
 5    unsafe {
 6        gl::UseProgram(program);
 7
 8        gl::DispatchCompute(
 9            size as gl::types::GLuint,
10            1,
11            1,
12        );
13
14        gl::MemoryBarrier(gl::ALL_BARRIER_BITS);
15    }
16    
17    // [...]
18}

Part 8: Read back data from the GPU’s shader storage buffer objects (SSBO)

 1fn main() {
 2    // [...]
 3
 4    // Read back from the GPU
 5    let output_data = read_ssbo_data(output_ssbo, size);
 6
 7    println!("[*] Output: {:?}", output_data);
 8    
 9    // [...]
10}
11
12fn read_ssbo_data(buffer: gl::types::GLuint, size: usize) -> Vec<u32> {
13    let mut data: Vec<u32> = Vec::with_capacity(size);
14
15    unsafe {
16        gl::BindBuffer(gl::SHADER_STORAGE_BUFFER, buffer);
17
18
19        let data_ptr =
20            gl::MapBuffer(gl::SHADER_STORAGE_BUFFER, gl::READ_ONLY) as *const u32;
21        if data_ptr.is_null() {
22            panic!("Failed to map buffer");
23        }
24
25        ptr::copy_nonoverlapping(
26            data_ptr,
27            data.as_mut_ptr(),
28            size,
29        );
30        data.set_len(size);
31
32        gl::UnmapBuffer(gl::SHADER_STORAGE_BUFFER);
33        gl::BindBuffer(gl::SHADER_STORAGE_BUFFER, 0);
34    }
35
36    data
37}

Part 9: Teardown OpenGL

 1fn main() {
 2    // [...]
 3
 4    // Teardown
 5    unsafe {
 6        gl::DeleteProgram(program);
 7        gl::DeleteShader(shader);
 8        gl::DeleteBuffers(1, &input_1_ssbo);
 9        gl::DeleteBuffers(1, &input_2_ssbo);
10        gl::DeleteBuffers(1, &output_ssbo);
11    }
12    
13    // [...]
14}

Part 10: Running the Code

$ cargo run
[*] Output: [164, 336, 516, 704, 900, 1104, 1316, 1536, 1764, 2000]

Resources

• Learn OpenGL
• Rust port of Learn OpenGL
• compute shaders in opengl 4.3
• Shader Storage Buffer Object
• Shader Buffers - Uniform & Shader Storage Buffer Objects
• Data Types (GLSL)
• Introduction - GPU Computing (Learn OpenGL)

From time to time I was working on an online-multiplayer web game, called Prompt Battler, which you can play here.

It is still in an early-development state, but you can find the code now here.

I am working on my first side project using Elixir and Phoenix. I was writing some tests and encountered a flaky unit test (randomly fails or passes). In this test, I wanted to make a case differentiation, but today I learned (TIL) in Elixir, the = behaves a bit differently. It is not an assignment operator; it is the match operator.

Pattern matching

When you use =, Elixir tries to match the value on the right side of the = with the pattern on the left side. If the pattern on the left does not match the value on the right, an error is raised.

1x = 42      # Matches and binds the value 42 to the variable x
2{a, b} = {1, 2}  # Matches and binds 1 to a, and 2 to b
3[head | tail] = [1, 2, 3]  # Matches and binds head to 1, tail to [2, 3]

The pin operator ^ is used to assert that a variable on the left side of the pattern should match the value on the right side. It prevents the introduction of a new variable with the same name.

1x = 42
2^x = 42  # Matches because the value on the right matches the existing variable x

Without the pin operator ^, the second line would introduce a new variable named x rather than asserting that it should match the existing x.

The Code

In the given code snippet, we have a example test case that involves generating a new game response. The response includes information about the winner and loser of the game, identified by their respective user IDs.

 1defmodule ExampleTest do
 2  use ExUnit.Case
 3
 4  def get_new_game_response() do
 5    random_number = :rand.uniform(10)
 6
 7    if random_number <= 5 do
 8      %{
 9        "winner" => %{"user_id" => "foo"},
10        "looser" => %{"user_id" => "bar"}
11      }
12    else
13      %{
14        "winner" => %{"user_id" => "bar"},
15        "looser" => %{"user_id" => "foo"}
16      }
17    end
18  end
19
20  test "new game response flacky" do
21    player_1 = "foo"
22    player_2 = "bar"
23
24    response = get_new_game_response()
25
26    case response["winner"]["user_id"] do
27      player_1 ->
28        assert response["looser"]["user_id"] =~ player_2
29
30      player_2 ->
31        assert response["looser"]["user_id"] =~ player_1
32
33      _ ->
34        raise "Unexpected user_id"
35    end
36  end
37
38  test "new game response with pin operator" do
39    player_1 = "foo"
40    player_2 = "bar"
41
42    response = get_new_game_response()
43
44    case response["winner"]["user_id"] do
45      ^player_1 ->
46        assert response["looser"]["user_id"] =~ player_2
47
48      ^player_2 ->
49        assert response["looser"]["user_id"] =~ player_1
50
51      _ ->
52        raise "Unexpected user_id"
53    end
54  end
55end

Pattern Matching Without Pin Operator

Let’s look at the original version of the test case without using the pin operator:

 1case response["winner"]["user_id"] do
 2  player_1 ->
 3    assert response["looser"]["user_id"] =~ player_2
 4
 5  player_2 ->
 6    assert response["looser"]["user_id"] =~ player_1
 7
 8  _ ->
 9    raise "Unexpected user_id"
10end

In this version, we attempt to match the user IDs of the winners (player_1 and player_2) and take corresponding actions. However, this code has a subtle issue that might not be immediately apparent.

A new variable binding occurs as this variable is used more than once in the same pattern. In our original code, both player_1 and player_2 are used as variables in the case pattern. So always the first case is executed, which randomly fails if player_2 is the winner but passes if not.

Using the Pin Operator

The pin operator (^) in Elixir is used to enforce a match against an existing variable’s value, preventing variable rebinding. Let’s revisit the modified test case that includes the pin operator:

 1case response["winner"]["user_id"] do
 2  ^player_1 ->
 3    assert response["looser"]["user_id"] =~ player_2
 4
 5  ^player_2 ->
 6    assert response["looser"]["user_id"] =~ player_1
 7
 8  _ ->
 9    raise "Unexpected user_id"
10end

By using the pin operator, we explicitly state that we want to match the value of player_1 and player_2 against the corresponding user IDs without introducing new bindings. This prevents any unintentional variable rebinding and ensures that our pattern matching behaves as expected.

Resources

• What is the “pin” operator for, and are Elixir variables mutable?
• Understand the pin “^" operator in Elixir

I am implementing a chat service using websockets based on Rust with Actix. Actix provides numerous examples, including a websocket-chat example, which I used as a reference for my implementation. However, I encountered difficulties while attempting to add my first unit test, following the testing documentation. Unfortunately, I couldn’t get it to work as I consistently received a response status of 400.

The Solution

After thorough debugging, I discovered that I was missing some headers, specifically:

• Upgrade: websocket
• Connection: Upgrade
• Sec-Websocket-Key: <my-websocket-key>
• Sec-WEbSocket-Version: 13

More information about these headers can be found on Websocket’s Wikipedia page.

Unlike in production scenarios, the value for Sec-Websocket-Key does not matter as long as the header exists.

In addition, I initially asserted 200 OK using assert!(resp.status().is_success());, following the documentation. However, a successful response when connecting to the WebSocket is 101 Switching Protocols, which I now verify using assert_eq!(resp.status(), 101);.

The Code

  1mod server;
  2mod session;
  3
  4use actix::{Actor, Addr, StreamHandler};
  5use actix_web::middleware::Logger;
  6use actix_web::{web, App, Error, HttpRequest, HttpResponse, HttpServer};
  7use actix_web_actors::ws;
  8
  9struct MyWs;
 10
 11impl Actor for MyWs {
 12    type Context = ws::WebsocketContext<Self>;
 13}
 14
 15impl StreamHandler<Result<ws::Message, ws::ProtocolError>> for MyWs {
 16    fn handle(&mut self, msg: Result<ws::Message, ws::ProtocolError>, ctx: &mut Self::Context) {
 17        match msg {
 18            Ok(ws::Message::Ping(msg)) => ctx.pong(&msg),
 19            Ok(ws::Message::Text(text)) => ctx.text(text),
 20            Ok(ws::Message::Binary(bin)) => ctx.binary(bin),
 21            _ => (),
 22        }
 23    }
 24}
 25
 26async fn chat_route(
 27    req: HttpRequest,
 28    stream: web::Payload,
 29    srv: web::Data<Addr<server::ChatServer>>,
 30) -> Result<HttpResponse, Error> {
 31    ws::start(
 32        session::WsChatSession {
 33            id: 0,
 34            room: Vec::new(),
 35            addr: srv.get_ref().clone(),
 36        },
 37        &req,
 38        stream,
 39    )
 40}
 41
 42#[actix_web::main]
 43async fn main() -> std::io::Result<()> {
 44    env_logger::init_from_env(env_logger::Env::new().default_filter_or("info"));
 45
 46    let server = server::ChatServer::new().start();
 47
 48    log::info!("starting HTTP server at http://localhost:8080");
 49
 50    HttpServer::new(move || {
 51        App::new()
 52            .app_data(web::Data::new(server.clone()))
 53            .route("/ws/", web::get().to(chat_route))
 54            .wrap(Logger::default())
 55    })
 56    .bind(("127.0.0.1", 8080))?
 57    .run()
 58    .await
 59}
 60
 61#[cfg(test)]
 62mod tests {
 63    use super::*;
 64
 65    use actix_web::body::MessageBody;
 66    use actix_web::dev::{ServiceFactory, ServiceRequest, ServiceResponse};
 67    use actix_web::{
 68        http::header::ContentType, http::header::HeaderValue, http::header::CONNECTION,
 69        http::header::SEC_WEBSOCKET_KEY, http::header::SEC_WEBSOCKET_VERSION,
 70        http::header::UPGRADE, test,
 71    };
 72
 73    fn setup() -> App<
 74        impl ServiceFactory<
 75            ServiceRequest,
 76            Response = ServiceResponse<impl MessageBody>,
 77            Config = (),
 78            InitError = (),
 79            Error = Error,
 80        >,
 81    > {
 82        env_logger::init_from_env(env_logger::Env::new().default_filter_or("info"));
 83
 84        let server = server::ChatServer::new().start();
 85
 86        App::new()
 87            .app_data(web::Data::new(server.clone()))
 88            .route("/ws/", web::get().to(chat_route))
 89            .wrap(Logger::default())
 90    }
 91
 92    #[actix_web::test]
 93    async fn test_unauthorized_without_authorization_header() {
 94        let app = test::init_service(setup()).await;
 95
 96        let req = test::TestRequest::get()
 97            .uri("/ws/")
 98            .insert_header((UPGRADE, HeaderValue::from_static("websocket")))
 99            .insert_header((CONNECTION, HeaderValue::from_static("Upgrade")))
100            .insert_header((SEC_WEBSOCKET_VERSION, HeaderValue::from_static("13")))
101            .insert_header((
102                SEC_WEBSOCKET_KEY,
103                HeaderValue::from_static("does-not-matter-for-testing"),
104            ))
105            .insert_header(ContentType::plaintext())
106            .to_request();
107
108        let resp = test::call_service(&app, req).await;
109
110        assert!(resp.status().is_informational());
111        assert_eq!(resp.status(), 101);
112    }
113}

Lessons Learned

Return App from a function

The return type is as follow:

1App<
2        impl ServiceFactory<
3            ServiceRequest,
4            Response = ServiceResponse<impl MessageBody>,
5            Config = (),
6            InitError = (),
7            Error = Error,
8        >,
9    >

Taken from here: How can I return App from a function / why is AppEntry private?, which links to the test code here.

Unhide test output

By default, the Rust test suite hides output from test execution, which prevented me from debugging this. To display all output, you need to run it as follows: cargo test -- --nocapture

As explained in the display options section in The Cargo Book.

I migrated to Pipenv some time ago and being pretty happy about this.

Outdated article

For every Python project I set up a virtualenv and handle my requirements with pip-tools. It is a command line tool to keep pip-based packages fresh and pinns them. It is my favorite tool to handle Pyhon requirements. Using pip-tools you basically define your requirements in a *.infile and when you run pip-compile it will create a *.txtrequirements files with all dependencies (and all underlying dependencies) pinned. And using pip-sync it will install all packages definied in a requirements file but also uninstall any other package which is not defined.

In most cases I split my requirements files into four categories: base, dev, tests and production. When developing I often have all requirements installed, in my CI only base and tests requirements and on production only base and production requirements.

So over time I approximately came to the following script, that lies as update.sh in the root of my Python projects. On a fresh virtualenv you first need to run pip install pip-tools, change the file permissions chmod 744 update.sh and then the following script can be used.

What it simply does when you run ./update.sh compile it will use pip-tools to create your pinned requirements files.

And then you can install the different combinations with ./update.sh sync-all for all packages, ./update.sh sync-tests for base and test and ./update.sh sync-dev for base, test and dev.

There are also commands which do not require pip-toolsto be installed, these use pipto install all needed dependencies. This is useful for CI environments or production where you install the dependencies only once and dont want unneded extra tools to be installed. You can run ./update sync-ciin the CI to install base and test or ./update sync-productionto install base and production requirements.

 1#!/bin/bash
 2
 3function help {
 4    echo "usage: ./update arguments"
 5    echo "  compile                 Upgrades the pinned version of all packages."
 6    echo "  sync-all                Update your virtual environment of all packages."
 7    echo "  sync-dev                Update your virtual environment for base, dev and test."
 8    echo "  sync-tests              Update your virtual environment for base and test."
 9    echo "  sync-ci                 Update your virtual environment for base and test."
10    echo "  sync-production         Update your virtual environment for base and production."
11    echo "  upgrade                 Upgrades pip, setuptools and pip-tools."
12}
13
14if [ $# -eq  0 ]; then
15    help
16    exit 1
17fi
18
19if [ "$1" == "sync-production" ]; then
20    pip install --no-cache-dir -r requirements/production.txt
21    exit 0
22elif [ "$1" == "sync-ci" ]; then
23    pip install --no-cache-dir -r requirements/tests.txt
24    exit 0
25elif [ "$1" == "upgrade" ]; then
26    pip install --upgrade pip setuptools pip-tools
27    exit 0
28fi
29
30if ! pip list | grep "pip-tools" > /dev/null ; then
31    echo "This action requires pip-tools. Run 'pip install pip-tools' or './update.sh upgrade'."
32    help
33    exit 1
34fi
35
36if [ "$1" == "sync-all" ]; then
37    pip-sync requirements/base.txt requirements/dev.txt requirements/tests.txt requirements/production.txt
38    exit 0
39
40elif [ "$1" == "sync-dev" ]; then
41    pip-sync requirements/base.txt requirements/dev.txt requirements/tests.txt
42    exit 0
43
44elif [ "$1" == "sync-tests" ]; then
45    pip-sync requirements/base.txt requirements/tests.txt
46    exit 0
47
48elif [ "$1" == "compile" ]; then
49    pip-compile --upgrade requirements/base.in -o requirements/base.txt
50    pip-compile --upgrade requirements/base.in requirements/tests.in requirements/dev.in -o requirements/dev.txt
51    pip-compile --upgrade requirements/base.in requirements/tests.in -o requirements/tests.txt
52    pip-compile --upgrade requirements/base.in requirements/production.in -o requirements/production.txt
53
54else
55    help
56    exit 1
57fi

References

• pip-tools

You are using gqlgen for building GraphQL servers in your Go projects. But your queryResolver or mutationResolver rely on some context value. For example as you are using gin and add values to your request context with middlewares. Therefore you need a way to modify the context of the gqlgen client in your tests.

An example resolver code could look like the following.

1func (r *mutationResolver) CreateCategory(ctx context.Context, title string) (*model.Category, error) {
2    currentUser := ctx.Value("User").(db.User)
3
4    // ...
5
6    return result, nil
7}

The solution

The following code shows you how to add values to context when testing gqlgen resolvers with the client.

 1// Add param variables to context of client request
 2// The gqlgen client accepts functional options as the following one
 3func addContext(user db.User) client.Option {
 4    return func(bd *client.Request) {
 5        ctx = context.WithValue(ctx, "User", user)
 6        bd.HTTP = bd.HTTP.WithContext(ctx)
 7    }
 8}
 9
10// Example tests
11func TestMutationResolver(t *testing.T) {
12    // Create the gqlgen client for testing
13    h := handler.NewDefaultServer(generated.NewExecutableSchema(generated.Config{Resolvers: &Resolver{}}))
14    c := client.New(h)
15
16    // Example test
17    t.Run("Create category", func(t *testing.T) {
18        // struct that saves the response of the mutation below
19        var resp struct {
20            CreateCategory struct{ Title string }
21        }
22
23        // Create the variable that is passed to addContext
24        user := db.User{}
25
26        // Call the resolver via the client and modify the context via functional options
27        c.MustPost(`
28            mutation($title: String!) {
29                createCategory(title:$title) {
30                    title
31                }
32            }`,
33            &resp,
34            client.Var("title", category.Title),
35            addContext(user),
36        )
37
38        // an example assertion
39        assert.Equal(t, "Awesome blogs", resp.CreateCategory.Title)
40    })
41}

References

• Awesome blog post about functional options in Go
• gqlgen is a Go library for building GraphQL servers
• Gin is a web framework written in Go

On some project I inserted a new column called position so the entries are orderable by the users. On this project users can create categories which have e.g. a title and may have a super/parent category and are owned by the user who created them.

The problem was that I wanted to have an initial alphabetical ordering of the categories and some rules had to apply. For the sorting only categories of the same user are considered and only categories that have a common super category.

The table looked similar to the following, however note that I have left out some columns and changed the data types (e.g. int as id instead of uuid) to make it more redeable.

The solution

1update categories
2	set position=r.row_id
3from (select c.id, row_number() over (partition by c.super_category_id, c.user_ref order by c.title asc) as row_id from categories c) as r
4where categories.id = r.id;

This sql query was the solution and would result in a table as seen below.

The inner select query divides the query’s result into partions. partition by c.super_category_id, c.user_ref order by c.title asc divides the query’s result set into partions, every partion has a unique super_category_id and user_ref and is in itself ordered alphabetically.

Using row_numer() we get the row number within the partion for every column and update the position column accordingly in the table.

References

• Update a column based on row_number()
• SQL PARTITION BY

Request guards represents some validation policy and is one of Rocket’s most powerful instruments. You can read more about guards in the docs. For example you can use guards to add authentication to handlers, there is even a nice example in the api doc. However, for simplification the valid api key is hard coded in the example. But we dont want this in a real project, so the problem is that we want to have a configurable api key.

The solution

One way to have a configurable api key is by using Rocket’s configuration abilities. You define a Rocket.toml that is used to configure Rocket but you can also add additional configuration parameters, like api_token. This file is committed so you cannot have your production secrets in it. In production you would overwrite your secrets with ROCKET_env variables, as these have a higher priority than the Rocket.tomlfile.

Then you can use AdHoc::config() to store a typed config in managed state.

And then retrieve it with request.rocket().state::<Config>().

The following code shows this in action based on the example taken from Rocket’s docs.

 1// src/main.rs
 2#[macro_use] extern crate rocket;
 3
 4use rocket::http::Status;
 5use rocket::request::{Outcome, Request, FromRequest};
 6use rocket::State;
 7use rocket::fairing::AdHoc;
 8use serde::Deserialize;
 9
10#[derive(Deserialize)]
11struct Config {
12    api_key: String,
13}
14
15struct ApiKey<'r>(&'r str);
16
17#[derive(Debug)]
18enum ApiKeyError {
19    Missing,
20    Invalid,
21}
22
23#[rocket::async_trait]
24impl<'r> FromRequest<'r> for ApiKey<'r> {
25    type Error = ApiKeyError;
26
27    async fn from_request(req: &'r Request<'_>) -> Outcome<Self, Self::Error> {
28
29        // Retrieve the config state like this
30        let config = req.rocket().state::<Config>().unwrap();
31
32        fn is_valid(key: &str, api_key: &str) -> bool {
33            key == api_key
34        }
35
36        match req.headers().get_one("Authorization") {
37            None => Outcome::Failure((Status::Unauthorized, ApiKeyError::Missing)),
38            Some(key) if is_valid(key, &config.api_key) => Outcome::Success(ApiKey(key)),
39            Some(_) => Outcome::Failure((Status::Unauthorized, ApiKeyError::Invalid)),
40        }
41    }
42}
43
44#[get("/")]
45async fn index(_config: State<'_, Config>, _key: ApiKey<'_>) -> &'static str {
46    "Hello, world!"
47}
48
49fn rocket() -> rocket::Rocket {
50    let rocket = rocket::ignite();
51    let figment = rocket.figment();
52
53    let _config: Config = figment.extract().expect("config");
54
55    // Store the typed config in managed state
56    rocket
57        .mount("/", routes![index])
58        .attach(AdHoc::config::<Config>())
59}
60
61#[rocket::main]
62async fn main() {
63    rocket()
64        .launch()
65        .await;
66}

 1# Cargo.toml
 2[package]
 3name = "example"
 4version = "0.1.0"
 5edition = "2018"
 6
 7[dependencies]
 8# Need latest version of Rocket (min. 5.0.0) as guard with async_trait is used
 9rocket = { git = "https://github.com/SergioBenitez/Rocket", version = "0.5.0-dev" }
10serde = { version = "1.0", features = ["derive"] }

1# Rocket.toml
2[default]
3api_key = "secret-api-key"
4
5[debug]
6api_key = "secret-api-key"
7
8[release]
9api_key = "secret-api-key"

References

• Rocket
• Rocket’s Request Guards
• Rocket’s Request Guard Example
• Rockets’s Docs for Configuration

I needed to use the 0.5.0-dev version of Rocket as I wanted to use the new async features. As this version is not yet published you need to point Cargo to a git repository. However, I needed some functionality of rocket_contrib, the Uuuid, but I was very confused on how to add it to the dependencies correctly. As I am currently very new on Rust, Rocket and the documentation did not properly explained it. I got the following error when compiling:

the trait `FromFormField<'_>` is not implemented for `rocket_contrib::uuid::Uuid`

At this point I was very confused as Uuid implements the FromFormField according to the docs. But the problem was that I did not properly configured my Cargo.toml

The solution

The correct Cargo.toml would be the following

1[package]
2name = "example"
3version = "0.1.0"
4edition = "2018"
5
6[dependencies]
7rocket = { git = "https://github.com/SergioBenitez/Rocket", version = "0.5.0-dev" }
8rocket_contrib = { git = "https://github.com/SergioBenitez/Rocket", version = "0.5.0-dev", default-features = false, features = ["uuid"] }

Or the following would also be correct

 1[package]
 2name = "example"
 3version = "0.1.0"
 4edition = "2018"
 5
 6[dependencies]
 7rocket = { git = "https://github.com/SergioBenitez/Rocket", version = "0.5.0-dev" }
 8
 9[dependencies.rocket_contrib]
10git = "https://github.com/SergioBenitez/Rocket"
11version = "0.5.0-dev"
12default-features = false
13features = ["uuid"]

References

• GitHub Issue
• Rust Book - Specifying Dependencies

At first we create a custom jupyter lab docker image. Create a file called Dockerfile and add the following content:

 1FROM python:3.8
 2
 3EXPOSE 8888
 4
 5RUN pip install --no-cache-dir jupyterlab \
 6    && useradd -ms /bin/bash lab
 7
 8USER lab
 9
10WORKDIR /home/lab
11
12CMD jupyter lab --ip=0.0.0.0

Now using this Dockerfile we build an image with sudo docker build -t jupyter-lab .

Run Image

You can start a jupyter lab instance with sudo docker run -it --name lab -p 8888:8888 -v "/change/local/path:/home/lab" jupyter-lab to which you can normally connect. This will start a container named lab, exposes jupyter lab to your host and mounts /change/local/path. If you want to automatically remove the container after stopping you can additionally add --rm to the command.

When you stopped the container and did not removed it you can restart it with sudo docker start lab -i.

Run /bin/bash

In a second terminal you can connect to the running container with sudo docker exec -it lab /bin/bash. This allows you to run aditional commands like pip install tensorflow.

 1import React from 'react';
 2
 3interface Props {
 4  greeting?: string
 5  planet: string
 6}
 7
 8const Greet = (props: Props) => {
 9  const { greeting, planet } = props;
10
11  return (
12    <p>
13      {`${greeting} from ${planet}`}
14    </p>
15  );
16};
17
18Greet.defaultProps = {
19  greeting: 'Hello',
20} as Partial<Props>;
21
22const component = <Greet planet="Earth" />

The as Partial<Props> is not necessary but it is a good practice as without it you could use wrong type as a default prop and TypeScript wouldn’t complain.

References

Resources

• GitHub Issue

Create a new app via create-react-app with the following command.

npx create-react-app my-browser-extension --template typescript

Or for Typescrip support run this command:

npx create-react-app my-browser-extension --template typescript

2. Setup the manifest

Create React App creates a public dir, delete everything except the manifets.json and index.html. The current manifest is for Web Apps, but we need a Web Extensions manifest. So replace the public/manifest.json content with for example the following one. An extension should have an icon, for the following manifest you would need to add a svg logo in the public dir.

 1{
 2  "manifest_version": 2,
 3  "name": "My-Browser-Extension",
 4  "version": "0.1",
 5  "description": "My browser extension",
 6  "homepage_url": "https://example.org",
 7  "icons": {
 8    "48": "logo.svg",
 9    "96": "logo.svg"
10  },
11  "permissions": [
12  ],
13  "web_accessible_resources": [
14  ]
15}

3. Setup Tailwind CSS

Install the Tailwind and its peer-dependencies using npm. As Create React App doesn’t support PostCSS 8 yet we need to install the Tailwind CSS v2.0 PostCSS 7 compatibility build.

1npm install tailwindcss@npm:@tailwindcss/postcss7-compat @tailwindcss/postcss7-compat postcss@^7 autoprefixer@^9 @craco/craco

We also installed @craco/craco, because Create React App doesn’t let you override the PostCSS configuration natively. Therefore we need to replace react-scripts with craco in the package.json like so:

 1{
 2    //...
 3    "scripts": {
 4    "start": "craco start",
 5    "build": "craco build",
 6    "test": "craco test",
 7    "eject": "react-scripts eject",
 8  },
 9  //...
10}

Then, generate your tailwind config via npx tailwindcss init. In the tailwind.config.js file you need to change the line purge: [] to purge: ['./src/**/*.{js,jsx,ts,tsx}', './public/index.html'].

Finally include Tailwind in your src/index.css

1/* src/index.css */
2@tailwind base;
3@tailwind components;
4@tailwind utilities;

4. Setup for production

Create React App embeds an inline script into index.html in the production build. But this would break the extension usage as it goes against its CSP (Content Security Policy), so we just disable it.

Change the build script in package.json to

1"build": "INLINE_RUNTIME_CHUNK=false craco build"

Setup for development

The goal is to create a live-reloading environment without ejecting.

First install the Webpack extension reloader plugin.

npm install -D webpack-extension-reloader

Then put the following script in scripts/watch.js and make it executable chmod +x ./scripts/watch.js.

 1#!/usr/bin/env node
 2
 3// Based on: https://mmazzarolo.com/blog/2019-10-19-browser-extension-development/
 4// Updated for: https://github.com/gsoft-inc/craco/blob/master/packages/craco/README.md#configuration
 5
 6// Force a "development" environment in watch mode
 7process.env.BABEL_ENV = "development";
 8process.env.NODE_ENV = "development";
 9
10const fs = require("fs-extra");
11const paths = require("react-scripts/config/paths");
12const webpack = require("webpack");
13const colors = require("colors/safe");
14const ExtensionReloader = require("webpack-extension-reloader");
15
16// use Craco to create the Webpack development config
17const { createWebpackDevConfig } = require("@craco/craco");
18const cracoConfig = require("../craco.config.js");
19const config = createWebpackDevConfig(cracoConfig);
20
21// The classic webpack-dev-server can't be used to develop browser extensions,
22// so we remove the "webpackHotDevClient" from the config "entry" point.
23config.entry = !Array.isArray(config.entry) ? config.entry : config.entry.filter(function(entry) {
24  return !entry.includes("webpackHotDevClient");
25});
26
27// Edit the Webpack config by setting the output directory to "./build".
28config.output.path = paths.appBuild;
29paths.publicUrl = paths.appBuild + "/";
30
31// Add the webpack-extension-reloader plugin to the Webpack config.
32// It notifies and reloads the extension on code changes.
33config.plugins.push(new ExtensionReloader());
34
35// Start Webpack in watch mode.
36const compiler = webpack(config);
37const watcher = compiler.watch({}, function(err) {
38  if (err) {
39    console.error(err);
40  } else {
41    // Every time Webpack finishes recompiling copy all the assets of the
42    // "public" dir in the "build" dir (except for the index.html)
43    fs.copySync(paths.appPublic, paths.appBuild, {
44      dereference: true,
45      filter: file => file !== paths.appHtml
46    });
47    // Report on console the succesfull build
48    console.clear();
49    console.info(colors.green("Compiled successfully!"));
50    console.info("Built at", new Date().toLocaleTimeString());
51    console.info();
52    console.info("Note that the development build is not optimized.");
53    console.info("To create a production build, use yarn build.");
54  }
55});

Finally, add a watch script to your package.json.

1"watch": "./scripts/watch.js"

5. Start development

You can now replace the created example with your html that is styled with Tailwind CSS. To develop your extension with live-reloading just run npm run watch or npm run build for production.

To load an extension in Firefox, open about:debugging, click This Firefox then click Load Temporary Add-on... and open your build/manifest.json file.

References

This blog article is mainly based on Matteo’s article Developing a browser extension with Create React App. But as we replaced react-scripts with cargo, we need to make a few adjustments here.

Resources

• Developing a browser extension with Create React App
• Install Tailwind CSS with Create React App
• Your first extension

You can easily get the latest free version via npm or yarn.

1$ npm install --save @fortawesome/fontawesome-free
2$ yarn add @fortawesome/fontawesome-free

Include the css file

Include the css file from node_modules into your _app.js file.

1import '../node_modules/@fortawesome/fontawesome-free/css/all.min.css';

Use it

Now you can use fontawesome via the icon class, e.g. fas fa-dog.

1<i className="fas fa-dog" />

Using components with known vulnerabilities is one of the top ten web application security risks of the famous OWASP Top Ten list. It is easy to forget to update your dependencies of your projects if this is a manual process. I am working in a small team (3 people) and we have to manage/maintain and program multiple services. For example some of our services are programmed in Python. All requirements we are using are pinned to a specific version, so every time we build our docker image the same packages are installed. However, if a service is completed or no more work is currently required its pretty easy to forget to update the pinned versions in the requirements file. If this would be the case our services may have known vulnerabilities that were already fixed and the chances of a successful attack would be higher.

The solution

However, this process can be automated using GitLab and there great CI/CD integration. GitLab’s pipelines let you automate steps like builds, tests, and deployments. GitLab pipelines are normally run based on certain conditions met, e.g. if you push your code. However, you can also schedule pipelines at specific intervals. For example every Monday at 5 a.m. or once a day.

In your projects sidebar navigate to CI / CD > Schedules.

Then click on New schedule button and fill out the form and save it. I choose the master branch as my target branch and scheduled the pipeline for every Monday at 5 a.m.

When the scheduled pipeline is run I want to create a merge request if any package could be updated.

 1# .gitlab-ci.yml
 2bot:
 3  # only run if this is a scheduled pipeline
 4  only:
 5    - schedules
 6
 7  image: python:3.8
 8
 9  script:
10    - cd $CI_PROJECT_DIR
11    # Checkout to a new branch so we push the changes to a new branch
12    - git checkout -b auto-update-$(date '+%Y-%m-%d')
13    - apt-get install -qy curl
14    # We are using pip-tools to handle/pin our dependencies
15    - pip install pip-tools
16    - chmod 700 update.sh
17    # This script updates my requirements file.
18    # This can be customized however you want to fit your needs.
19    - ./update.sh
20    # If any line was changed then commit the changes, push them and create a merge request
21    - |
22      lines=$(git status -s | wc -l)
23      if [ $lines -gt 0 ];then
24        echo "committing"
25        git config --global user.name "${AUTO_COMMITTER_NAME}"
26        git config --global user.email "${AUTO_COMMITTER_EMAIL}"
27        git add .
28        git commit -m "auto-update-$(date '+%Y-%m-%d')"
29        git push "https://${GITLAB_USER_NAME}:${GIT_PUSH_TOKEN}@${CI_REPOSITORY_URL#*@}"
30        BODY="{\"id\": \"${CI_PROJECT_ID}\", \"source_branch\": \"auto-update-$(date '+%Y-%m-%d')\", \"target_branch\": \"master\", \"remove_source_branch\": true, \"title\": \"Auto Update $(date '+%Y-%m-%d')\"}"
31        curl -X POST "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/merge_requests" \
32                --header "PRIVATE-TOKEN:${GIT_PUSH_TOKEN}" \
33                --header "Content-Type: application/json" \
34                --data "${BODY}";
35        echo "Opened a new merge request: \"Auto Update $(date '+%Y-%m-%d')\" and assigned to you";
36        exit;
37      else
38        echo "no change, nothing to commit"
39      fi      
40
41test:
42  stage: test
43
44  # dont run if this is a scheduled pipeline
45  except:
46    - schedules
47  ...

The update.sh script is responsible to update the requirements file and pin the newest version of each package. We are using pip-tools to handle/pin our dependencies. It’s a great tool. You only need to define you packages you want to install and it automatically pinns the newest version of this packages and the packages dependencies in a requirements.txt file. It makes it super easy to use pinned packages.

For example your update.sh script could be as simple as the following. This will generate up-to date pinned requirements files.

1#!/bin/bash
2# update.sh
3pip-compile --upgrade requirements/*.in -o requirements/all.txt
4pip-compile --upgrade requirements/base.in requirements/tests.in -o requirements/tests.txt
5pip-compile --upgrade requirements/base.in requirements/production.in -o requirements/production.txt

Finally, we just need to define some environment variables. Go to Settings -> CI/CD and add three variables:

1. AUTO_COMMITTER_EMAIL which is the email that should be assigned to the commit.
2. AUTO_COMMITTER_NAME which is the name that should be assigned to the commit.
3. GIT_PUSH_TOKEN an api token, needed for authorization to push the commit and open a merge request. To create an token navigate to User Settings -> Access Tokens and create a personal access token with api scope. The access token needs to be generated by the person who created the schedule pipeline. You may create a custom bot user, so no one has to use her own access token.

If you want to test it, navigate back to CI / CD > Schedules and click the play button to run the pipeline.

Now you have a scheduled pipeline that runs every Monday at 5 a.m., checks whether any dependency can be updated and if so it creates a merge request. This can then be approved by you and merge into the master branch. Make sure you have a good test coverage so the new versions of your dependencies do not break your code.

If you have any further questions, I made a mistake or something else you can contact me. Either write a comment or contact me via email.

So, taking into account the global communication networks, data centers and personal computers you could approximate the needed electricity consumption. Turns out for the year 2012 this was calculated to be about 930 TWh [EINS Consortium et al. 2013]. This correspondence to 4.7% of the total worldwide electricity consumption in 2012 [EINS Consortium et al. 2013]. This translates to about 530 Mt of CO2, which corresponds to 1.7% of the worldwide total amount of greenhouse gases in 2012 [Gelenbe et al.].

However, this calculation did not include TVs, smart phones, audio devices, manufacturing and dispose of equipment [EINS Consortium et al. 2013]. If you add the electricity consumption for manufacturing, smartphones and TVs the total electricity consumption almost doubles to about 1,815 TWh per year [Corcoran et al.]. This then correspondence to 9,17% of the worldwide electricity consumption and 3.3% of the worldwide greenhouse emissions in 2012.

Before 2012 growing rate of 6.6% per year in the collective electricity consumption of communication networks, data centers and personal computers has been seen [EINS Consortium et al. 2013]. Therefore, the today’s electricity consumption and CO2 emissions are probably significantly underestimated by above numbers. Additionally, the current deep learning and IoT hype are not reflected in this numbers as well.

We all use the internet every day. For example me, most of the time, I sit in front of a screen. And almost any of the things I do require a working internet connection. Today we all use the possibilities of the world wide web to socialize, learn, entertainment purpose, work and heavily rely on it. However, what we often don’t see or don’t think of are the causalities. The increased usage of information and communication technologies does not come at zero costs. The needed electricity to power and manufacture all these servers, devices and things has grown enormously. But also dependencies on rare materials and violations of human rights and environmental destruction to gather the needed materials. Facing a climate crisis right know we need to rethink, grow awareness and change it, fast. Therefore, I want to dedicate a mini series on this topic.

References

[Corcoran et al.] Corcoran, P. & Andrae, A. Emerging Trends in Electricity Consumption for Consumer ICT. 57.

[EINS Consortium et al. 2013] EINS Consortium and others. Overview of ICT energy consumption (D8. 1). (2013).

[Gelenbe et al.] Gelenbe, E. & Caseau, Y. The impact of information technology on energy consumption and carbon emissions. Ubiquity 2015, 1–15 (2015).

Last year Google announced to work on Manifest V3 (MV3) and a lot of voices raised that Google wants to kill adblockers. This resulted in a back and forth between the Chrome developers and the adblocking community.

MV3 was released as a preview/alpha on October 31st, 2019 and a stable release is planned for 2020.

Currently the Chrome browsers offers extensions the possibility to observe and block network requests via the webRequest API. However, the Chrome developers raised concerns about the security and performance impact by this practice. Therefore, the new declarativeNetRequest API was proposed to handle blocking of network requests. With this new API extensions should be able to tell Chrome what to do with a given request. However, the amount of entries are limited. First to 30.000 entries but then was raised to 150.000. As EasyList, a popular blocking list used in all adblockers, already hash 68,000 entries on its own. And some adblockers also uses more advanced techniques than simple blacklisting.[Manifest V3] [Easy List]

Comments from the adblock community

Use privacy/performance as Trojan arguments to rationalize reducing user agency over what all bloated web sites throw at people’s user agents. That new declarativeNetRequest API seriously reduces what blockers can do, to the point they will become distinguishable only by their UI, not their capabilities. As a user, I personally wouldn’t accept browsing the world wild web without the advanced features in uBO, I find this unthinkable. [source]

– Raymond Hill (gorhill) (developer of uBlock Origin and uMatrix)

There is no way to transpose either either dynamic filtering, dynamic URL filtering, per-site/per-scope switch logic (let’s refer to all these as “dynamic filtering”), into static filters. Dynamic filtering logic requires an arbitrary amount of block/allow rules overriding other block/allow rules based on specificity. There is no concept of specificity in static filtering – and even more, there is no concept of dynamic filtering rules relinquishing filtering to static filters (dynamic filtering’s noop rules).

And there I was recently testing how uBO handled over half a million network filters with the 3rd-gen HNTrie… [source]

– Raymond Hill (gorhill) (developer of uBlock Origin and uMatrix)

For the sake of not just the present, but also the healthy and open web ecosystem of the future, we need to make sure that we allow APIs that continue to foster innovation. Our very serious concern is that the proposed changes in manifest V3, especially with the proposed changes to webRequest and the current implementation of declarativeNetRequest might very well do the opposite. […] Most modern ad blockers are already quite efficient and are constantly looking to improve performance further. We would encourage the Chromium team to feel free to engage more with the ad blocking developer community regarding manifest V3. We would be happy to provide more technical feedback, suggestions and support. source

– Feedback from various contributors/supporters of adblock community

The new API is not in itself a bad thing, but it becomes a bad thing when it’s the only option because it lacks the flexibility that the Web Requests API provides. source

– Jeremy Tillman (president of Ghostery)

I actually don’t care about the hard-coded limit on blacklists because I use a whitelist, but I need contextual information which the Declarative API’s stated purpose is keeping away from extensions. source

– Giorgio Maone (developer of NoScript)

Google’s arguments

Google brings up two major arguments for introducing the declarativeNetRequest API, (1) performance and (2) security.

Performance

Currently via the webRequest API, Chrome sends all the data in a network request to the listening extensions. These extensions than tells Chrome what to do with the request, like allow it, block it or send it with some modifications. Using the declarativeNetRequest API extensions would register a limited amount of rules that tell Chrome on certain types of requests what to do. As evaluation to allow/disallow a request is now performed by the browser, no serializing, shuttling the inter-process messages to the extensions and waiting for a decision for all request data is needed anymore. [Chromium blog post]

Ghostery has made two studies regarding performance influence by adblockers. The first study Adblockers Performance Study measured the decision times per request for several popular adblockers like uBlock Origin, Ghostery and AdBlock Plus. The measurements returned a sub-millisecond median decision time per request, what compared to the actual time a request needs to be send/returned to a server, is negligible. Additionally, with the upcoming WebAssembly technology adblockers could reduce decision time even more.

But the second study, The Tracker Tax, is even more interesting. They simply compared the loading time of the top 500 websites with and without blocking trackers. It turns out that the average page load time with all trackers blocked is about 2 times faster than when no trackers are blocked. This is a significant performance increase by using adblockers.

And this makes sense. The time for a request to be serialized, passed to the extensions and then waiting for the decision is already very fast. This all happens on your local machine managed by a highly optimized piece of software, your browser. But then the request needs to be sent/travel to the server, be processed by the server and then sent/travel back to you. This is the part that takes most of the time. Every request that can be avoided significantly lowers the loading time. Modern websites today connect to so many different services, significantly slowing down the page load time. I deeply recommend you to install uMatrix or NoScript and then visit some of your favorite websites. This would give you an impression of many web services are contacted on each visit.

Security

Google states that 42% of malicious extensions use the Web Request API. As every network request is passed to the extensions a malicious extension may has access to sensitive data.

However,

The webRequest API is not going to be fully removed as part of Manifest V3. In particular, there are currently no planned changes to the observational capabilities of webRequest (i.e., anything that does not modify the request). [source]

– Simeon Vincent (Google Extensions Developer Advocate)

So malicious extensions are still able to grab sensitive data but adblockers are limtied in there functionality.

Additionaly, adblockers help preventing being infected by Malvertising, a technique to use online advertising to spread malware.

Impact on electricity consumption

This may be a bit off topic, but I found it too interesting to ignore.

Just recently a study, Energy Conservation with Open Source Ad Blockers, was published that evaluated the impact of adblockers regarding electricity consumption. Every request made consumes some amount of electricity, because it needs to be transferred to a destination, being processed and a response sent back. So every request avoided actually reduce the electricity consumption. It was found out that using adblockers has a significantly impact/potential not only on reducing electricity consumption.

Energy Conservation with Open Source Ad Blockers - Abstract

Internet-related electricity consumption is rising rapidly as global Internet users spend more than 6.5 h per day online. Open source ad blockers have the potential to reduce the time and thus electricity spent using computers by eliminating ads during Internet browsing and video streaming. In this study, three open source ad blockers are tested against a no-ad blocker control. Page load time is recorded for browsing a representative selection of the globally most-accessed websites, and the time spent watching ads on videos is quantified for both trending and non-trending content. The results show that page load time dropped 11% with AdBlock+, 22.2% with Privacy Badger, and 28.5% with uBlock Origin. Thus, uBlock Origin has the potential to save the average global Internet user more than 100 h annually. The energy conserved if everyone in the United States used the open source ad blocker would save over 36 Americans lives per year if it were to offset coal-fired electricity generated-based pollution. In the United States, if all Internet users enabled Privacy Badger on their computers, Americans would save more than $91 million annually. Globally, uBlock Origin could save consumers more than $1.8 billion/year. Open source ad blockers are a potentially effective technology for energy conservation. [source]

– Pearce, J.M. Energy Conservation with Open Source Ad Blockers. Technologies 2020, 8, 18.

Conclusion

After Chrome announced Manifest V3 a lot of voices raised that Google wants to kill adblockers. However, Google always negated this accuse and has put forward technical arguments to defend there decisions. But as shown the two major arguments performance and security do not even hold true on a basic consideration.

By the way in 2018 over 85% of Alphabets (owner of Google) total revenue was made from ad revenues Alphabet Annual Report. Besides all the products Google has created they are still mainly an ad company, it is still there core business model. Therefore, any blocking of ads has an influence on there profit. Adblockers directly affect Google’s profit. And even if Google has helped improving the web it is still a company that is responsible for there stackholders and maximizing profit. Therefore, adblockers are a big business risk for Google as Google said by themselve

New and existing technologies could affect our ability to customize ads and/or could block ads online, which would harm our business.

Technologies have been developed to make customizable ads more difficult or to block the display of ads altogether and some providers of online services have integrated technologies that could potentially impair the core functionality of third-party digital advertising. Most of our Google revenues are derived from fees paid to us in connection with the display of ads online. As a result, such technologies and tools could adversely affect our operating results. [source] –

Google Chrome now dominates the browser market with about 68 percent of the global desktop internet browser market share. Additionally, many browser are based on Chrome as Microsoft’s Edge or Opera. This gives Google the power to force new APIs and overall shape the future of the internet as no other company. Not even speaking about there supremacy as a search engine or additional services. But being an ad and stackholder company, Google’s vision of the future may not be always the best for us and a free and open web for all of us.

So maybe it’s finally time to ditch Chrome…

Create a new virtual environment or enter an already existing one and install the ipykernel. The following code creates a new one, to enter an existing one use workon my_env.

1$ mkvirtualenv my_venv
2$ pip install ipykernel
3$ python -m ipykernel install --user --name my_venv

Then you can start your juypter lab or notebook in another terminal.

1$ jupyter lab
2$ jupyter notebook

Follow the generated link and choose the kernel called my_env.

If you need to install a python package, e.g. matplotlib, just install it into your virtual environment to be able to use it in jupyter.

1$ pip install matplotlib

Additional

If you are using ArchLinux, btw. it just got 18 years old (initial release was 11 March 2002), you can install Jupyter Notebook or Jupyter Lab via pacman.

1$ sudo pacman -S jupyterlab
2$ sudo pacman -S jupyter-notebook