Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2705 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.7107. Min Act: -0.0001

Data Index: 330725 (C4 (Web Text))

Max Activating Token Index: 770

Text #1

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.7219. Min Act: -0.0000

Data Index: 33168 (C4 (Web Text))

Max Activating Token Index: 47

Text #2

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.6986. Min Act: -0.0000

Data Index: 182711 (C4 (Web Text))

Max Activating Token Index: 101

Text #3

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.6825. Min Act: -0.0001

Data Index: 1388326 (Python Code)

Max Activating Token Index: 233

Text #4

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.6764. Min Act: -0.0001

Data Index: 521897 (C4 (Web Text))

Max Activating Token Index: 489

Text #5

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.6796. Min Act: -0.0000

Data Index: 177169 (C4 (Web Text))

Max Activating Token Index: 823

Text #6

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.6830. Min Act: -0.0001

Data Index: 1138114 (C4 (Web Text))

Max Activating Token Index: 915

Text #7

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.6219. Min Act: -0.0000

Data Index: 761407 (C4 (Web Text))

Max Activating Token Index: 906

Text #8

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.5837. Min Act: -0.0000

Data Index: 58049 (C4 (Web Text))

Max Activating Token Index: 12

Text #9

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.6065. Min Act: -0.0001

Data Index: 341672 (C4 (Web Text))

Max Activating Token Index: 103

Text #10

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.6043. Min Act: -0.0001

Data Index: 689978 (C4 (Web Text))

Max Activating Token Index: 966

Text #11

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.5717. Min Act: -0.0001

Data Index: 29965 (C4 (Web Text))

Max Activating Token Index: 422

Text #12

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.5646. Min Act: -0.0000

Data Index: 724020 (C4 (Web Text))

Max Activating Token Index: 105

Text #13

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.5719. Min Act: -0.0000

Data Index: 353901 (C4 (Web Text))

Max Activating Token Index: 454

Text #14

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.5699. Min Act: -0.0001

Data Index: 1521653 (Python Code)

Max Activating Token Index: 412

Text #15

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.5516. Min Act: -0.0001

Data Index: 1495996 (Python Code)

Max Activating Token Index: 137

Text #16

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.5578. Min Act: -0.0001

Data Index: 481418 (C4 (Web Text))

Max Activating Token Index: 511

Text #17

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.5579. Min Act: -0.0000

Data Index: 451870 (C4 (Web Text))

Max Activating Token Index: 77

Text #18

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.5559. Min Act: -0.0001

Data Index: 1177593 (C4 (Web Text))

Max Activating Token Index: 124

Text #19

Max Range: 0.7219. Min Range: -0.7219

Max Act: 0.5369. Min Act: -0.0000

Data Index: 679080 (C4 (Web Text))

Max Activating Token Index: 51