Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2943 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1672. Min Range: -2.1672

Max Act: 2.0508. Min Act: -0.0000

Data Index: 1211713 (C4 (Web Text))

Max Activating Token Index: 5

Text #1

Max Range: 2.1672. Min Range: -2.1672

Max Act: 2.1270. Min Act: -0.0000

Data Index: 1085297 (C4 (Web Text))

Max Activating Token Index: 774

Text #2

Max Range: 2.1672. Min Range: -2.1672

Max Act: 2.1672. Min Act: -0.0000

Data Index: 1267793 (C4 (Web Text))

Max Activating Token Index: 789

Text #3

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.9654. Min Act: -0.0000

Data Index: 1312346 (C4 (Web Text))

Max Activating Token Index: 977

Text #4

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.9581. Min Act: -0.0000

Data Index: 903819 (C4 (Web Text))

Max Activating Token Index: 2

Text #5

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.9261. Min Act: -0.0000

Data Index: 532050 (C4 (Web Text))

Max Activating Token Index: 270

Text #6

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.9040. Min Act: -0.0000

Data Index: 871775 (C4 (Web Text))

Max Activating Token Index: 211

Text #7

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.8970. Min Act: -0.0000

Data Index: 503400 (C4 (Web Text))

Max Activating Token Index: 565

Text #8

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.8808. Min Act: -0.0000

Data Index: 1249499 (C4 (Web Text))

Max Activating Token Index: 3

Text #9

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.7564. Min Act: -0.0000

Data Index: 954186 (C4 (Web Text))

Max Activating Token Index: 189

Text #10

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.8906. Min Act: -0.0000

Data Index: 703938 (C4 (Web Text))

Max Activating Token Index: 842

Text #11

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.7697. Min Act: -0.0000

Data Index: 364014 (C4 (Web Text))

Max Activating Token Index: 722

Text #12

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.6720. Min Act: -0.0000

Data Index: 705375 (C4 (Web Text))

Max Activating Token Index: 365

Text #13

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.8425. Min Act: -0.0000

Data Index: 1001568 (C4 (Web Text))

Max Activating Token Index: 852

Text #14

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.8345. Min Act: -0.0000

Data Index: 651192 (C4 (Web Text))

Max Activating Token Index: 658

Text #15

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.7272. Min Act: -0.0000

Data Index: 1130679 (C4 (Web Text))

Max Activating Token Index: 416

Text #16

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.6278. Min Act: -0.0000

Data Index: 1212214 (C4 (Web Text))

Max Activating Token Index: 821

Text #17

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.7193. Min Act: -0.0000

Data Index: 1287501 (C4 (Web Text))

Max Activating Token Index: 601

Text #18

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.6050. Min Act: -0.0000

Data Index: 791028 (C4 (Web Text))

Max Activating Token Index: 978

Text #19

Max Range: 2.1672. Min Range: -2.1672

Max Act: 1.7038. Min Act: -0.0000

Data Index: 878191 (C4 (Web Text))

Max Activating Token Index: 378