Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2232 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.2841. Min Range: -1.2841

Max Act: 1.2841. Min Act: -0.0000

Data Index: 1146920 (C4 (Web Text))

Max Activating Token Index: 282

Text #1

Max Range: 1.2841. Min Range: -1.2841

Max Act: 1.0840. Min Act: -0.0000

Data Index: 90242 (C4 (Web Text))

Max Activating Token Index: 482

Text #2

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.9942. Min Act: -0.0000

Data Index: 116718 (C4 (Web Text))

Max Activating Token Index: 367

Text #3

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.9581. Min Act: -0.0000

Data Index: 511264 (C4 (Web Text))

Max Activating Token Index: 237

Text #4

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.9375. Min Act: -0.0000

Data Index: 1146772 (C4 (Web Text))

Max Activating Token Index: 588

Text #5

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.9411. Min Act: -0.0000

Data Index: 185926 (C4 (Web Text))

Max Activating Token Index: 585

Text #6

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.8635. Min Act: -0.0000

Data Index: 1255318 (C4 (Web Text))

Max Activating Token Index: 823

Text #7

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.8640. Min Act: -0.0000

Data Index: 499169 (C4 (Web Text))

Max Activating Token Index: 486

Text #8

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.9145. Min Act: -0.0000

Data Index: 1303571 (C4 (Web Text))

Max Activating Token Index: 1001

Text #9

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.9718. Min Act: -0.0000

Data Index: 376888 (C4 (Web Text))

Max Activating Token Index: 123

Text #10

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.9719. Min Act: -0.0000

Data Index: 1015868 (C4 (Web Text))

Max Activating Token Index: 725

Text #11

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.9087. Min Act: -0.0000

Data Index: 217088 (C4 (Web Text))

Max Activating Token Index: 974

Text #12

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.8977. Min Act: -0.0000

Data Index: 53162 (C4 (Web Text))

Max Activating Token Index: 136

Text #13

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.9475. Min Act: -0.0000

Data Index: 789682 (C4 (Web Text))

Max Activating Token Index: 336

Text #14

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.8673. Min Act: -0.0000

Data Index: 153264 (C4 (Web Text))

Max Activating Token Index: 381

Text #15

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.8487. Min Act: -0.0000

Data Index: 1080200 (C4 (Web Text))

Max Activating Token Index: 705

Text #16

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.8731. Min Act: -0.0000

Data Index: 649676 (C4 (Web Text))

Max Activating Token Index: 977

Text #17

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.8182. Min Act: -0.0000

Data Index: 523137 (C4 (Web Text))

Max Activating Token Index: 237

Text #18

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.8228. Min Act: -0.0000

Data Index: 491672 (C4 (Web Text))

Max Activating Token Index: 319

Text #19

Max Range: 1.2841. Min Range: -1.2841

Max Act: 0.8613. Min Act: -0.0000

Data Index: 971577 (C4 (Web Text))

Max Activating Token Index: 248