Ultimate Guide to HTTPS: Enhancing Web Security in 10 steps

Leave a Comment / HLD / By

What is HTTPS protocol?

HTTPS stands for Hyper Text Transfer Protocol Secure. It is an extension of the HTTP protocol used primarily for secure communication over a computer network. In HTTPS, the communication protocol is encrypted using Transport Layer Security (TLS), or formerly, its predecessor, Secure Sockets Layer (SSL). This means that the data exchanged between the web browser and the website is encrypted to prevent eavesdropping and tampering.

HTTP is fundamentally designed as a request-response protocol, where a client (usually a web browser or app) must send a request to a server to receive any information or service. This design is intrinsic to its operation; without a request, the server does not send any data, as it needs to know what resources the client is asking for and how to appropriately respond.

However, there are scenarios where it might seem like HTTP is operating without an explicit request from the user:

  1. Automatic Requests: Websites often include scripts that automatically make HTTP requests in the background without direct user action, for tasks like fetching updates, sending heartbeat signals to maintain sessions, or loading additional resources dynamically (AJAX).
  2. Push Technology: Although standard HTTP doesn’t initiate from the server, some web technologies allow servers to send updates to the client proactively. For example, technologies such as WebSockets and HTTP/2’s server push feature can enable servers to send data to the client as soon as it’s available, without waiting for a new request for each piece of data.
  3. Preloading: Some browsers use preloading techniques to fetch certain links that the user might click next. This is based on predictive algorithms but still initially relies on an HTTP request made by the client as part of loading the current page.
  4. Persistent Connections: With HTTP/1.1 and later, persistent connections allow multiple requests and responses to flow back and forth over a single connection without reestablishing it each time, but each interaction still begins with a request from the client.

Despite these extensions and optimizations, the basic principle of HTTP as a request-response protocol remains: the server requires an initial request to know what action to perform and what data to send.

How does HTTPS protocol work?

The detailed internal workings of HTTPS involve several key steps to ensure secure and encrypted communication between a client (such as a web browser) and a server:

Step 1. URL Request

The process begins when a user enters an HTTPS URL in their browser or clicks on an HTTPS link.

Step 2. DNS Lookup

The browser performs a DNS lookup to translate the domain name into an IP address.

Step 3. TCP Connection

Similar to HTTP, HTTPS first establishes a TCP connection using the standard three-way handshake (SYN, SYN-ACK, ACK).

The establishment of a TCP (Transmission Control Protocol) connection involves a specific sequence known as the three-way handshake. This process is designed to create a reliable connection between a client and a server. Here’s a detailed explanation of each step in the three-way handshake:

Step 3.1: SYN

  • Initiation by the Client: The client begins the handshake process by sending a SYN (synchronize) packet to the server. This packet includes a sequence number (SEQ), which is typically a random number, that serves as an initial sequence number for the messages the client will send.
  • Purpose: The SYN flag in the packet indicates that it is a connection request, and the sequence number is used to synchronize the sequence numbers used during the TCP session.

Step 3.2: SYN-ACK

  • Server Response: Upon receiving the SYN packet, the server responds with a SYN-ACK (synchronize-acknowledge) packet. This packet has two main components:
  • ACK Number: This is set to one more than the sequence number received from the client, acknowledging the client’s SYN packet.
  • Server’s SYN: The packet also includes the server’s own SYN, with a sequence number chosen by the server, which will be used as the initial sequence number for the messages the server sends.
  • Purpose: The SYN-ACK packet confirms to the client that the server received its initial SYN packet and is also ready to establish a connection.

Step 3.3: ACK

  • Completion by the Client: Finally, the client sends an ACK (acknowledge) packet back to the server. This packet’s acknowledgment number is set to one more than the sequence number of the SYN received from the server.
  • Purpose: This last packet confirms to the server that its SYN-ACK packet was received by the client, and now both parties have acknowledged each other’s sequence numbers and readiness to communicate.

Connection Established

  • Once this three-way handshake is completed, both the client and server have agreed on initial sequence numbers and have acknowledged the connection setup. The TCP connection is now established, and data can begin flowing between the two.

Reliable and Ordered Communication

  • Reliability and Order: TCP ensures that data sent over a network is delivered in order and without errors. It uses sequence numbers to order packets correctly and acknowledgments to confirm receipt. If packets are lost, delayed, or arrive out of order, TCP detects these issues and can retransmit data or reorder packets as needed.

This three-step handshake is a fundamental aspect of TCP’s design, providing a reliable way to establish and manage connections that ensures data integrity and order between devices communicating over a network.

Step 4. TLS Handshake

Once the TCP connection is established, the TLS handshake begins, which is the core of HTTPS’s security features:

Step 4.1 ClientHello:

The client sends a “ClientHello” message that includes the TLS version it supports, the cipher suites (encryption algorithms) it supports, and a randomly generated number.

Step 4.2 ServerHello

The server responds with a “ServerHello” message, selecting the TLS version and cipher suite from the list provided by the client. The server also sends a randomly generated number.

Step 4.3 Server Certificate:

The server sends its digital certificate to the client. This certificate contains the server’s public key and is signed by a trusted certificate authority (CA). The client uses this certificate to verify the server’s identity.

Step 4.4 Key Exchange

Depending on the chosen cipher suite, the key exchange process occurs. Often, the client uses the server’s public key to encrypt a pre-master secret and sends it to the server, which decrypts it using its private key.

Step 4.5 Client and Server Finished

Both parties compute a master secret from the pre-master secret and exchanged random numbers, and then they send a finished message, encrypted with a key derived from the master secret.

Step 5. Encrypted Session

With the TLS handshake complete, both the client and server have established shared secret keys used to encrypt and decrypt the data transmitted during the session.

Step 6. HTTPS Request

The client now sends an HTTPS request to the server. This request is encrypted with the established keys:

HTTPS Request Components

  1. Start Line
    • Method: The HTTP method (e.g., GET, POST, PUT, DELETE, etc.) indicates the action to be performed on the specified resource.
    • Request URI: The Uniform Resource Identifier specifies the resource being requested.
    • HTTP Version: Specifies the version of HTTP being used (e.g., HTTP/1.1 or HTTP/2).
  2. Headers
    • Request Headers: Include metadata about the request and client, such as:
      • Host: The domain name of the server.
      • User-Agent: Information about the client software making the request.
      • Accept: The content types that the client can handle.
      • Accept-Language: Preferred languages for the response.
      • Cookie: Any cookies that the client needs to send to the server.
      • Authorization: Credentials for authenticating the client to the server.
      • Content-Type: The type of data in the body (for POST and PUT requests).
      • Content-Length: The length of the request body.
  3. Body (optional, not used with all methods)
    • Contains the data sent to the server. This might include form data, file uploads, or JSON/XML payload, depending on the context and the method used.

Step 7. Server Processing

The server decrypts the HTTPS request using the symmetric key, processes it, and prepares an HTTPS response.

Step 8. HTTPS Response

The server sends an encrypted HTTP response back to the client:

HTTPS Response Components

  1. Status Line
    • HTTP Version: Indicates the HTTP version used by the server.
    • Status Code: A three-digit number that indicates the outcome of the request (e.g., 200 OK, 404 Not Found).
    • Status Text: A textual phrase associated with the status code, providing a human-readable explanation.
  2. Headers
    • Response Headers: Include metadata about the response and server, such as:
      • Content-Type: The type of the data in the body of the response.
      • Content-Length: The length of the response body.
      • Set-Cookie: Cookies that the server wishes to store on the client.
      • Cache-Control: Directives for caching mechanisms.
      • Date: The date and time the message was sent.
      • Server: Information about the server software.
  3. Body (optional)
    • The actual data returned by the server. This could be HTML, JSON, XML, binary data like images or files, etc.

Step 9. Closing the Connection

After the transaction, the connection can be closed, or if kept open for further exchanges, will continue to be secured by TLS.

Step 10. Content Display

The browser decrypts the HTTPS response and renders the content for the user.

Additional Security Checks

Throughout this process, security checks, like verifying the server’s certificate against a list of trusted CAs and ensuring the integrity of the data received, are continuously performed to prevent attacks like eavesdropping, tampering, or forgery.

This flow ensures that all data transmitted between the client and the server is securely encrypted and that both parties are authenticated, significantly enhancing the security of data transfer over the Internet compared to HTTP.

Scenarios where HTTPS will not be a useful protocol

HTTP and HTTPS are incredibly versatile, but there are scenarios where these protocols might not be the most efficient or suitable choice. While designing a system, we need to understand all the nuances to make the correct design choices. Here are 20 unique scenarios where other protocols could be more appropriate:

  1. Real-Time Gaming: For multiplayer online games requiring real-time interaction, UDP (User Datagram Protocol) is often preferred due to lower latency compared to HTTP.
  2. Voice Over IP (VoIP): Protocols like RTP (Real-Time Protocol) are used for voice communication over the internet because they support continuous transmission required for smooth voice communication.
  3. Live Video Streaming: While HTTP/HTTPS can be used for streaming, protocols like RTSP (Real-Time Streaming Protocol) or RTP are better suited for live video feeds due to their lower latency and better control over streaming sessions.
  4. Large Scale File Transfers: For transferring large files, FTP (File Transfer Protocol) or SFTP (Secure File Transfer Protocol) might be more effective as they offer features specifically tailored for file management and transfer.
  5. Email Transmission: SMTP (Simple Mail Transfer Protocol) is used for sending emails, and IMAP or POP for retrieving emails, as they are protocols specifically designed for email communications.
  6. IoT Device Communication: Protocols like MQTT (Message Queuing Telemetry Transport) or CoAP (Constrained Application Protocol) are designed for the minimal bandwidth and power requirements typical of IoT devices.
  7. Secure Shell Access (SSH): SSH is used for secure remote login and command execution, which is not something HTTP/HTTPS is designed to handle.
  8. Network Management: SNMP (Simple Network Management Protocol) is used for network device management and monitoring, which requires capabilities outside the scope of HTTP/HTTPS.
  9. Binary Data Transfer in Microservices: gRPC uses HTTP/2 for transport but is specifically designed for the efficient communication between microservices, using binary data instead of text-based formats like HTTP.
  10. Database Synchronization: Database-specific protocols like replication protocols used by SQL or NoSQL databases are optimized for data consistency and synchronization.
  11. Version Control Systems: Systems like Git use their own protocols for data transfer and version control, which are more efficient for the types of data and operations involved.
  12. Blockchain Transactions: Blockchain platforms often use custom protocols designed for maintaining and verifying the ledger of transactions, which are not feasible with HTTP/HTTPS.
  13. Financial Transactions in Banking Systems: Banking systems often use ISO 8583 protocol for transaction messages, which provides a more secure and direct approach than HTTP/HTTPS.
  14. Satellite Communication: Specialized communication protocols designed for the unique latency and error-correction requirements of satellite links.
  15. Machine-to-Machine (M2M) Communication: Protocols designed for automation and industrial systems, like Modbus or Fieldbus, which are not based on HTTP/HTTPS.
  16. Automotive Telematics: Protocols designed for vehicle communications, which require real-time data exchange and may use vehicle-specific protocols.
  17. Home Automation Systems: Protocols like Zigbee or Z-Wave, designed for home automation, offer direct device communication that HTTP/HTTPS cannot provide.
  18. Sensor Networks: Wireless sensor networks often use lightweight, custom protocols optimized for low power consumption and data transmission efficiency.
  19. Remote Desktop Services: Protocols like RDP (Remote Desktop Protocol) provide functionalities for remote desktop access that HTTP/HTTPS cannot achieve.
  20. VPN Traffic: VPN protocols such as OpenVPN or IPsec are used to create secure tunnels for encrypted communication, a different use case than what HTTP/HTTPS are designed for.

Each of these scenarios involves requirements that HTTP/HTTPS are not designed to handle, whether it be low latency, binary data transmission, continuous data stream, or specialized device communication.

General guide on how to understand when not to use HTTPS

Deciding when not to use HTTP or HTTPS involves considering the specific requirements and constraints of the application you’re working with. Here are some key factors to guide you in determining when HTTP/HTTPS might not be the optimal choice:

1. Real-Time Performance Requirements

If your application requires real-time performance with minimal latency, such as in online gaming, voice over IP (VoIP), or live video streaming, HTTP/HTTPS may not be suitable due to the overhead of establishing connections and the TCP protocol’s nature which prioritizes reliability over latency.

2. Efficiency in Data Transmission

HTTP/HTTPS can be inefficient for scenarios needing high-performance data transfer, especially with binary data. Protocols like gRPC or even raw TCP sockets might be more suitable because they allow more control over the data formats and transmission mechanics.

3. Protocol-Specific Features Needed

Some applications require features that HTTP/HTTPS do not provide, such as:

  • Multicast communications often used in software distribution or streaming media.
  • Session management in a stateful context, where protocols like WebSocket might be a better fit.
  • File transfer operations (e.g., FTP or SFTP) which include abilities to pause, resume, or manage files.

4. Low Resource Environments

In environments where resources are constrained, such as IoT devices, HTTP/HTTPS might consume too much power and bandwidth. Protocols like MQTT or CoAP are designed to be lightweight and efficient for such environments.

5. Security Requirements Beyond What HTTPS Offers

While HTTPS adds a layer of security to HTTP, it may not be sufficient for all cases. For example, applications requiring end-to-end encryption controlled entirely by the end users might benefit from additional or different security protocols.

6. Legacy Systems Compatibility

If you’re dealing with legacy systems that do not support HTTP/HTTPS or modern encryption standards, you might need to use older or different protocols that those systems support.

7. Large-Scale File Transfers

HTTP/HTTPS isn’t the best for large-scale file transfers compared to dedicated file transfer protocols like FTP, which offer more robust features for managing and optimizing such transfers.

Decision-Making Tips

  • Assess Application Needs: Understand what your application needs in terms of speed, data handling, and security.
  • Consider the Environment: Take into account where your application will run—whether on servers, desktops, mobile devices, or embedded systems.
  • Review Industry Standards: Look at what protocols are standard for your industry or for the type of application you’re developing.

Using HTTP/HTTPS is often the default for web-based applications due to their wide support and built-in security features with HTTPS. However, always evaluate whether another protocol might better meet your specific requirements or offer more efficiency.

See more in

Leave a Comment

Your email address will not be published. Required fields are marked *