Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 799 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.2339. Min Act: -0.0000

Data Index: 1306167 (C4 (Web Text))

Max Activating Token Index: 649

Text #1

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.2206. Min Act: -0.0000

Data Index: 316944 (C4 (Web Text))

Max Activating Token Index: 875

Text #2

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1862. Min Act: -0.0000

Data Index: 139156 (C4 (Web Text))

Max Activating Token Index: 650

Text #3

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1629. Min Act: -0.0000

Data Index: 1113673 (C4 (Web Text))

Max Activating Token Index: 966

Text #4

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1522. Min Act: -0.0000

Data Index: 1231446 (C4 (Web Text))

Max Activating Token Index: 304

Text #5

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1736. Min Act: -0.0000

Data Index: 81246 (C4 (Web Text))

Max Activating Token Index: 344

Text #6

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1785. Min Act: -0.0000

Data Index: 143541 (C4 (Web Text))

Max Activating Token Index: 845

Text #7

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1591. Min Act: -0.0000

Data Index: 1223438 (C4 (Web Text))

Max Activating Token Index: 694

Text #8

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1628. Min Act: -0.0000

Data Index: 649729 (C4 (Web Text))

Max Activating Token Index: 588

Text #9

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1599. Min Act: -0.0000

Data Index: 905691 (C4 (Web Text))

Max Activating Token Index: 164

Text #10

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1477. Min Act: -0.0000

Data Index: 79891 (C4 (Web Text))

Max Activating Token Index: 206

Text #11

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1327. Min Act: -0.0000

Data Index: 908609 (C4 (Web Text))

Max Activating Token Index: 447

Text #12

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1401. Min Act: -0.0000

Data Index: 1336100 (C4 (Web Text))

Max Activating Token Index: 859

Text #13

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1331. Min Act: -0.0000

Data Index: 919601 (C4 (Web Text))

Max Activating Token Index: 478

Text #14

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1203. Min Act: -0.0000

Data Index: 1202134 (C4 (Web Text))

Max Activating Token Index: 212

Text #15

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1267. Min Act: -0.0000

Data Index: 1257252 (C4 (Web Text))

Max Activating Token Index: 481

Text #16

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1266. Min Act: -0.0000

Data Index: 34144 (C4 (Web Text))

Max Activating Token Index: 42

Text #17

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1277. Min Act: -0.0000

Data Index: 534110 (C4 (Web Text))

Max Activating Token Index: 371

Text #18

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1181. Min Act: -0.0000

Data Index: 1210333 (C4 (Web Text))

Max Activating Token Index: 893

Text #19

Max Range: 0.2339. Min Range: -0.2339

Max Act: 0.1314. Min Act: -0.0000

Data Index: 632882 (C4 (Web Text))

Max Activating Token Index: 900