To antialias points or lines, you need to enable antialiasing by calling glEnable() and passing in GL_POINT_SMOOTH or GL_LINE_SMOOTH, as appropriate. You can also provide a quality hint by calling glHint(). The hint parameter can be GL_FASTEST to indicate that the most efficient option should be chosen, GL_NICEST to indicate the highest quality option should be chosen, or GL_DONT_CARE to indicate no preference.
When antialiasing is enabled, OpenGL computes the fraction of each pixel that is covered by the point or line. The setting of the GL_LINE_SMOOTH and the GL_POINT_SMOOTH hints determine how accurate the calculation is when rendering lines and points, respectively. When the hint is set to GL_NICEST, a larger filter function may be applied causing more fragments to be generated and rendering to slow down.
If you are using RGBA rendering, OpenGL will set the alpha value according to the pixel coverage. You need to enable blending so that the incoming pixel fragment will be combined with the value already in the framebuffer, depending on the alpha value. You will probably want to set the blending factors to GL_SRC_ALPHA (source) and GL_ONE_MINUS_SRC_ALPHA (destination). You can also use GL_ONE for the destination factor to make lines a little brighter where they intersect.
Antialiasing in color index mode is trickier because you have to load the color map correctly to get primitive edges to blend with the background color. When antialiasing is enabled, the last four bits of the color index indicate the coverage value. Thus, you need to load sixteen contiguous colormap locations with a color ramp ranging from the background color to the object's color. This technique only works well when drawing wireframe images, where the lines and points typically need to be blended with a constant background. If the lines and/or points need to be blended with background polygons or images, RGBA rendering should be used.
You need to be careful when rendering antialiased lines and points with depth buffered primitives. You should draw the depth buffered primitives first and then draw the points and lines with the depth test still enabled but with depth buffer updates disabled. This way the points and lines will be correctly depth buffered against the rest of the geometry. This is a similar algorithm to that used for drawing a mixture of opaque and translucent surfaces with depth buffering. If antialiased lines are drawn with the normal depth buffering algorithm a halo artifact may be visible at the intersections of lines. This halo is a result of the antialiased lines being drawn several pixels wide with the pixels along the edges of the line having attenuated alpha values which can also occlude pixels with larger depth values (i.e., parts of other lines). When drawing antialiased lines it is often necessary to adjust the gamma of the monitor to get the best results.