Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1564 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.8953. Min Act: -0.0000

Data Index: 1316317 (C4 (Web Text))

Max Activating Token Index: 87

Text #1

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.9073. Min Act: -0.0000

Data Index: 421663 (C4 (Web Text))

Max Activating Token Index: 814

Text #2

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.7616. Min Act: -0.0000

Data Index: 452318 (C4 (Web Text))

Max Activating Token Index: 387

Text #3

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.6530. Min Act: -0.0000

Data Index: 562806 (C4 (Web Text))

Max Activating Token Index: 765

Text #4

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.8132. Min Act: -0.0000

Data Index: 153874 (C4 (Web Text))

Max Activating Token Index: 444

Text #5

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.4226. Min Act: -0.0000

Data Index: 52036 (C4 (Web Text))

Max Activating Token Index: 919

Text #6

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.4826. Min Act: -0.0000

Data Index: 952109 (C4 (Web Text))

Max Activating Token Index: 893

Text #7

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.4494. Min Act: -0.0000

Data Index: 371266 (C4 (Web Text))

Max Activating Token Index: 140

Text #8

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.2837. Min Act: 0.0003

Data Index: 1164388 (C4 (Web Text))

Max Activating Token Index: 864

Text #9

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.2908. Min Act: -0.0000

Data Index: 876064 (C4 (Web Text))

Max Activating Token Index: 690

Text #10

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.3765. Min Act: -0.0000

Data Index: 151087 (C4 (Web Text))

Max Activating Token Index: 345

Text #11

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.4927. Min Act: -0.0000

Data Index: 452130 (C4 (Web Text))

Max Activating Token Index: 728

Text #12

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.3569. Min Act: -0.0000

Data Index: 1348835 (C4 (Web Text))

Max Activating Token Index: 854

Text #13

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.1791. Min Act: -0.0000

Data Index: 262288 (C4 (Web Text))

Max Activating Token Index: 651

Text #14

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.3108. Min Act: -0.0000

Data Index: 613706 (C4 (Web Text))

Max Activating Token Index: 335

Text #15

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.2482. Min Act: -0.0000

Data Index: 94434 (C4 (Web Text))

Max Activating Token Index: 737

Text #16

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.1101. Min Act: -0.0000

Data Index: 797117 (C4 (Web Text))

Max Activating Token Index: 720

Text #17

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.0534. Min Act: -0.0000

Data Index: 639637 (C4 (Web Text))

Max Activating Token Index: 580

Text #18

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.1040. Min Act: -0.0000

Data Index: 319894 (C4 (Web Text))

Max Activating Token Index: 307

Text #19

Max Range: 1.9073. Min Range: -1.9073

Max Act: 1.1045. Min Act: -0.0000

Data Index: 933733 (C4 (Web Text))

Max Activating Token Index: 1017