Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3687 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.3467. Min Range: -2.3467

Max Act: 2.3467. Min Act: -0.0000

Data Index: 968464 (C4 (Web Text))

Max Activating Token Index: 877

Text #1

Max Range: 2.3467. Min Range: -2.3467

Max Act: 2.2825. Min Act: -0.0000

Data Index: 267224 (C4 (Web Text))

Max Activating Token Index: 65

Text #2

Max Range: 2.3467. Min Range: -2.3467

Max Act: 2.1700. Min Act: -0.0000

Data Index: 647735 (C4 (Web Text))

Max Activating Token Index: 965

Text #3

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.9666. Min Act: -0.0000

Data Index: 876459 (C4 (Web Text))

Max Activating Token Index: 399

Text #4

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.9996. Min Act: -0.0000

Data Index: 711705 (C4 (Web Text))

Max Activating Token Index: 99

Text #5

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.8410. Min Act: -0.0000

Data Index: 908593 (C4 (Web Text))

Max Activating Token Index: 865

Text #6

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.8842. Min Act: -0.0000

Data Index: 1110916 (C4 (Web Text))

Max Activating Token Index: 636

Text #7

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.8544. Min Act: -0.0000

Data Index: 409699 (C4 (Web Text))

Max Activating Token Index: 686

Text #8

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.9229. Min Act: -0.0000

Data Index: 1276747 (C4 (Web Text))

Max Activating Token Index: 346

Text #9

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.7512. Min Act: -0.0000

Data Index: 1352746 (C4 (Web Text))

Max Activating Token Index: 821

Text #10

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.8670. Min Act: -0.0000

Data Index: 427401 (C4 (Web Text))

Max Activating Token Index: 946

Text #11

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.7508. Min Act: -0.0000

Data Index: 997057 (C4 (Web Text))

Max Activating Token Index: 999

Text #12

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.8359. Min Act: -0.0000

Data Index: 1118227 (C4 (Web Text))

Max Activating Token Index: 860

Text #13

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.6355. Min Act: -0.0000

Data Index: 480530 (C4 (Web Text))

Max Activating Token Index: 817

Text #14

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.5994. Min Act: -0.0000

Data Index: 1052315 (C4 (Web Text))

Max Activating Token Index: 681

Text #15

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.6833. Min Act: -0.0000

Data Index: 1225773 (C4 (Web Text))

Max Activating Token Index: 49

Text #16

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.6752. Min Act: -0.0000

Data Index: 1343260 (C4 (Web Text))

Max Activating Token Index: 748

Text #17

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.6608. Min Act: -0.0000

Data Index: 1056407 (C4 (Web Text))

Max Activating Token Index: 378

Text #18

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.7592. Min Act: -0.0000

Data Index: 612258 (C4 (Web Text))

Max Activating Token Index: 408

Text #19

Max Range: 2.3467. Min Range: -2.3467

Max Act: 1.6436. Min Act: -0.0000

Data Index: 1269256 (C4 (Web Text))

Max Activating Token Index: 330