Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2439 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.4352. Min Act: -0.0000

Data Index: 412328 (C4 (Web Text))

Max Activating Token Index: 247

Text #1

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.4131. Min Act: -0.0000

Data Index: 33870 (C4 (Web Text))

Max Activating Token Index: 841

Text #2

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.3884. Min Act: -0.0000

Data Index: 277079 (C4 (Web Text))

Max Activating Token Index: 276

Text #3

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.3628. Min Act: -0.0001

Data Index: 1324363 (C4 (Web Text))

Max Activating Token Index: 869

Text #4

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.2453. Min Act: -0.0001

Data Index: 334694 (C4 (Web Text))

Max Activating Token Index: 381

Text #5

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.2160. Min Act: -0.0001

Data Index: 163110 (C4 (Web Text))

Max Activating Token Index: 21

Text #6

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.2140. Min Act: -0.0001

Data Index: 470422 (C4 (Web Text))

Max Activating Token Index: 627

Text #7

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.2069. Min Act: -0.0001

Data Index: 496995 (C4 (Web Text))

Max Activating Token Index: 686

Text #8

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.2111. Min Act: -0.0001

Data Index: 216569 (C4 (Web Text))

Max Activating Token Index: 999

Text #9

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.1960. Min Act: -0.0001

Data Index: 1358570 (C4 (Web Text))

Max Activating Token Index: 39

Text #10

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.1851. Min Act: -0.0001

Data Index: 376894 (C4 (Web Text))

Max Activating Token Index: 774

Text #11

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.1919. Min Act: -0.0001

Data Index: 1233509 (C4 (Web Text))

Max Activating Token Index: 118

Text #12

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.2219. Min Act: -0.0001

Data Index: 226851 (C4 (Web Text))

Max Activating Token Index: 776

Text #13

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.2164. Min Act: -0.0001

Data Index: 228090 (C4 (Web Text))

Max Activating Token Index: 160

Text #14

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.1691. Min Act: -0.0001

Data Index: 219717 (C4 (Web Text))

Max Activating Token Index: 871

Text #15

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.1580. Min Act: -0.0000

Data Index: 382816 (C4 (Web Text))

Max Activating Token Index: 122

Text #16

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.1528. Min Act: -0.0000

Data Index: 652361 (C4 (Web Text))

Max Activating Token Index: 390

Text #17

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.1329. Min Act: -0.0000

Data Index: 1391151 (Python Code)

Max Activating Token Index: 994

Text #18

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.1369. Min Act: -0.0000

Data Index: 1289457 (C4 (Web Text))

Max Activating Token Index: 181

Text #19

Max Range: 0.4352. Min Range: -0.4352

Max Act: 0.1626. Min Act: -0.0001

Data Index: 950635 (C4 (Web Text))

Max Activating Token Index: 651