Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4028 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.4145. Min Act: -0.0000

Data Index: 188963 (C4 (Web Text))

Max Activating Token Index: 44

Text #1

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.3835. Min Act: -0.0000

Data Index: 289432 (C4 (Web Text))

Max Activating Token Index: 1016

Text #2

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.3053. Min Act: -0.0000

Data Index: 451131 (C4 (Web Text))

Max Activating Token Index: 526

Text #3

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.2841. Min Act: -0.0000

Data Index: 159489 (C4 (Web Text))

Max Activating Token Index: 487

Text #4

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.2383. Min Act: -0.0000

Data Index: 860913 (C4 (Web Text))

Max Activating Token Index: 794

Text #5

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.2676. Min Act: -0.0000

Data Index: 923964 (C4 (Web Text))

Max Activating Token Index: 760

Text #6

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.2670. Min Act: -0.0000

Data Index: 790247 (C4 (Web Text))

Max Activating Token Index: 739

Text #7

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.2224. Min Act: -0.0000

Data Index: 309372 (C4 (Web Text))

Max Activating Token Index: 183

Text #8

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.2614. Min Act: -0.0000

Data Index: 1151537 (C4 (Web Text))

Max Activating Token Index: 677

Text #9

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.3347. Min Act: -0.0000

Data Index: 138927 (C4 (Web Text))

Max Activating Token Index: 919

Text #10

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.2187. Min Act: -0.0000

Data Index: 1282010 (C4 (Web Text))

Max Activating Token Index: 548

Text #11

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.3149. Min Act: -0.0000

Data Index: 1143151 (C4 (Web Text))

Max Activating Token Index: 804

Text #12

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.2468. Min Act: -0.0000

Data Index: 1118597 (C4 (Web Text))

Max Activating Token Index: 60

Text #13

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.2668. Min Act: -0.0000

Data Index: 1280708 (C4 (Web Text))

Max Activating Token Index: 900

Text #14

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.2711. Min Act: -0.0000

Data Index: 810298 (C4 (Web Text))

Max Activating Token Index: 892

Text #15

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.1878. Min Act: -0.0000

Data Index: 1274786 (C4 (Web Text))

Max Activating Token Index: 960

Text #16

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.2244. Min Act: -0.0000

Data Index: 809877 (C4 (Web Text))

Max Activating Token Index: 155

Text #17

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.1403. Min Act: -0.0000

Data Index: 854240 (C4 (Web Text))

Max Activating Token Index: 704

Text #18

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.1690. Min Act: -0.0000

Data Index: 1253710 (C4 (Web Text))

Max Activating Token Index: 842

Text #19

Max Range: 1.4145. Min Range: -1.4145

Max Act: 1.1995. Min Act: -0.0000

Data Index: 796421 (C4 (Web Text))

Max Activating Token Index: 958