Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3966 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.0541. Min Range: -2.0541

Max Act: 2.0317. Min Act: -0.0000

Data Index: 1245213 (C4 (Web Text))

Max Activating Token Index: 350

Text #1

Max Range: 2.0541. Min Range: -2.0541

Max Act: 2.0541. Min Act: -0.0000

Data Index: 391426 (C4 (Web Text))

Max Activating Token Index: 583

Text #2

Max Range: 2.0541. Min Range: -2.0541

Max Act: 2.0146. Min Act: -0.0000

Data Index: 425718 (C4 (Web Text))

Max Activating Token Index: 847

Text #3

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.6957. Min Act: -0.0000

Data Index: 1325755 (C4 (Web Text))

Max Activating Token Index: 953

Text #4

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.7332. Min Act: -0.0000

Data Index: 975925 (C4 (Web Text))

Max Activating Token Index: 893

Text #5

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.5446. Min Act: -0.0000

Data Index: 1204945 (C4 (Web Text))

Max Activating Token Index: 456

Text #6

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.6383. Min Act: -0.0000

Data Index: 846219 (C4 (Web Text))

Max Activating Token Index: 883

Text #7

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.6318. Min Act: -0.0000

Data Index: 31073 (C4 (Web Text))

Max Activating Token Index: 491

Text #8

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.6279. Min Act: -0.0000

Data Index: 869936 (C4 (Web Text))

Max Activating Token Index: 391

Text #9

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.5759. Min Act: -0.0000

Data Index: 246834 (C4 (Web Text))

Max Activating Token Index: 457

Text #10

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.5625. Min Act: -0.0000

Data Index: 393473 (C4 (Web Text))

Max Activating Token Index: 201

Text #11

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.4495. Min Act: -0.0000

Data Index: 1103527 (C4 (Web Text))

Max Activating Token Index: 948

Text #12

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.3651. Min Act: -0.0000

Data Index: 292163 (C4 (Web Text))

Max Activating Token Index: 768

Text #13

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.5281. Min Act: -0.0000

Data Index: 482620 (C4 (Web Text))

Max Activating Token Index: 251

Text #14

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.4744. Min Act: -0.0000

Data Index: 702624 (C4 (Web Text))

Max Activating Token Index: 447

Text #15

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.5431. Min Act: -0.0000

Data Index: 1119629 (C4 (Web Text))

Max Activating Token Index: 354

Text #16

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.6224. Min Act: -0.0000

Data Index: 1081472 (C4 (Web Text))

Max Activating Token Index: 281

Text #17

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.4365. Min Act: -0.0000

Data Index: 1002645 (C4 (Web Text))

Max Activating Token Index: 170

Text #18

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.4338. Min Act: -0.0000

Data Index: 1264875 (C4 (Web Text))

Max Activating Token Index: 906

Text #19

Max Range: 2.0541. Min Range: -2.0541

Max Act: 1.4791. Min Act: -0.0000

Data Index: 552600 (C4 (Web Text))

Max Activating Token Index: 591